[Paper Review] Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition

Peng Wang, Jun Wen+3 more|arXiv (Cornell University)|Nov 22, 2021

Human Pose and Action RecognitionComputer Science72 references58 citations

TL;DR

This paper proposes Contrast-Reconstruction Representation Learning (CRRL), a self-supervised framework for skeleton-based action recognition that jointly learns skeletal postures and motion dynamics by combining sequence reconstruction with contrastive learning on velocity sequences. By using knowledge distillation to transfer motion knowledge from a contrastive learner to a reconstruction autoencoder, CRRL achieves state-of-the-art performance on multiple benchmarks without requiring labeled data.

ABSTRACT

Skeleton-based action recognition is widely used in varied areas, e.g., surveillance and human-machine interaction. Existing models are mainly learned in a supervised manner, thus heavily depending on large-scale labeled data which could be infeasible when labels are prohibitively expensive. In this paper, we propose a novel Contrast-Reconstruction Representation Learning network (CRRL) that simultaneously captures postures and motion dynamics for unsupervised skeleton-based action recognition. It mainly consists of three parts: Sequence Reconstructor, Contrastive Motion Learner, and Information Fuser. The Sequence Reconstructor learns representation from skeleton coordinate sequence via reconstruction, thus the learned representation tends to focus on trivial postural coordinates and be hesitant in motion learning. To enhance the learning of motions, the Contrastive Motion Learner performs contrastive learning between the representations learned from coordinate sequence and additional velocity sequence, respectively. Finally, in the Information Fuser, we explore varied strategies to combine the Sequence Reconstructor and Contrastive Motion Learner, and propose to capture postures and motions simultaneously via a knowledge-distillation based fusion strategy that transfers the motion learning from the Contrastive Motion Learner to the Sequence Reconstructor. Experimental results on several benchmarks, i.e., NTU RGB+D 60, NTU RGB+D 120, CMU mocap, and NW-UCLA, demonstrate the promise of the proposed CRRL method by far outperforming state-of-the-art approaches.

Motivation & Objective

• To address the limitation of existing self-supervised skeleton-based action recognition methods that focus excessively on static postures and underfit motion dynamics.
• To improve unsupervised representation learning by incorporating explicit motion signals, such as velocity sequences, into the learning process.
• To enable joint modeling of posture and motion in a single representation through knowledge distillation from a contrastive motion learner to a reconstruction autoencoder.
• To validate the effectiveness of combining reconstruction and contrastive learning for self-supervised skeleton-based action recognition.

Proposed method

• The Sequence Reconstructor (SER) uses an encoder-decoder architecture to reconstruct raw skeleton coordinate sequences, learning spatial posture representations.
• The Contrastive Motion Learner (CML) performs contrastive learning between representations from coordinate sequences and velocity sequences to enhance motion dynamics learning.
• The Information Fuser (INF) applies knowledge distillation, where the query encoder of CML acts as a teacher to guide the SER encoder, transferring motion-aware features.
• The framework uses a momentum-updated key encoder and a non-updated key MLP to stabilize contrastive learning and improve mutual information.
• The model is trained end-to-end with a combination of reconstruction loss and contrastive loss, with temperature scaling and memory bank for positive and negative sample management.
• The final representation is derived from the student encoder in SER, which now captures both posture and motion due to distillation.

Experimental results

Research questions

• RQ1 Can incorporating velocity sequences into self-supervised skeleton-based action recognition improve motion dynamics learning compared to relying solely on coordinate reconstruction?
• RQ2 Does knowledge distillation from a contrastive motion learner to a reconstruction autoencoder effectively transfer motion knowledge while preserving posture representation?
• RQ3 How do hyperparameters such as temperature and negative sample count affect the performance of the contrastive learning module in CRRL?
• RQ4 Does the proposed CRRL framework outperform existing self-supervised methods on standard skeleton-based action recognition benchmarks?

Key findings

• CRRL achieves 73.8% accuracy on the NTU RGB+D 60 dataset under the cross-subject protocol, outperforming state-of-the-art methods.
• On the NTU RGB+D 120 dataset, CRRL achieves 67.6% accuracy, demonstrating strong generalization across larger action sets.
• The ablation study shows that using both forward and reverse reconstruction improves performance by 2.5% compared to unidirectional reconstruction.
• The use of a non-updated key MLP in the contrastive module improves performance by 1.5% compared to a momentum-updated key MLP.
• The optimal temperature hyperparameter τ is found to be 0.1, balancing softness of similarity distribution and contrastive learning effectiveness.
• The VGSR baseline, which jointly learns velocity and skeleton reconstruction, still underperforms CRRL, indicating the superiority of the proposed distillation-based fusion strategy.