[Paper Review] SimMC: Simple Masked Contrastive Learning of Skeleton Representations for Unsupervised Person Re-Identification

Haocong Rao, Chunyan Miao|arXiv (Cornell University)|Apr 21, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes SimMC, a simple masked contrastive learning framework for unsupervised person re-identification using 3D skeleton sequences. By leveraging masked prototype contrastive learning (MPC) to cluster representative features and masked intra-sequence contrastive learning (MIC) to exploit motion continuity, SimMC learns discriminative skeleton representations without labels, achieving state-of-the-art performance on multiple benchmarks, including a 42.5% top-1 accuracy on CASIA-B and a 5.6% improvement over SM-SGE.

ABSTRACT

Recent advances in skeleton-based person re-identification (re-ID) obtain impressive performance via either hand-crafted skeleton descriptors or skeleton representation learning with deep learning paradigms. However, they typically require skeletal pre-modeling and label information for training, which leads to limited applicability of these methods. In this paper, we focus on unsupervised skeleton-based person re-ID, and present a generic Simple Masked Contrastive learning (SimMC) framework to learn effective representations from unlabeled 3D skeletons for person re-ID. Specifically, to fully exploit skeleton features within each skeleton sequence, we first devise a masked prototype contrastive learning (MPC) scheme to cluster the most typical skeleton features (skeleton prototypes) from different subsequences randomly masked from raw sequences, and contrast the inherent similarity between skeleton features and different prototypes to learn discriminative skeleton representations without using any label. Then, considering that different subsequences within the same sequence usually enjoy strong correlations due to the nature of motion continuity, we propose the masked intra-sequence contrastive learning (MIC) to capture intra-sequence pattern consistency between subsequences, so as to encourage learning more effective skeleton representations for person re-ID. Extensive experiments validate that the proposed SimMC outperforms most state-of-the-art skeleton-based methods. We further show its scalability and efficiency in enhancing the performance of existing models. Our codes are available at https://github.com/Kali-Hac/SimMC.

Motivation & Objective

• To address the limitations of supervised and hand-crafted skeleton-based person re-ID methods that require expensive labeling and domain-specific modeling.
• To develop a generic, label-free framework that learns discriminative skeleton representations directly from raw 3D skeleton sequences.
• To improve representation learning by exploiting both inter-prototype similarity and intra-sequence motion consistency in unlabeled data.
• To enhance scalability and compatibility with existing models and RGB-based skeleton estimation pipelines.

Proposed method

• Proposes masked prototype contrastive learning (MPC) to cluster and contrast subsequence representations from randomly masked segments of raw skeleton sequences against learned skeleton prototypes.
• Uses instance-prototype contrastive loss to pull similar instances closer and push dissimilar ones apart, enabling self-supervised feature learning.
• Introduces masked intra-sequence contrastive learning (MIC) to capture pattern consistency between temporally adjacent subsequences, leveraging motion continuity.
• Employs a multi-layer perceptron (MLP) backbone with minimal architectural complexity to directly process raw 3D joint coordinates.
• Applies random masking strategies to augment training data and improve robustness to sequence variations.
• Fuses MPC and MIC losses with a learnable weight (λ) to jointly optimize discriminative and consistent representation learning.

Experimental results

Research questions

• RQ1 Can a self-supervised contrastive learning framework effectively learn discriminative skeleton representations for person re-identification without any human-annotated labels?
• RQ2 How can masked subsequence representations be used to discover and contrast with typical skeleton prototypes to improve feature learning?
• RQ3 To what extent does modeling intra-sequence motion consistency enhance the quality of learned skeleton representations?
• RQ4 How does the proposed SimMC framework compare to state-of-the-art supervised and self-supervised skeleton-based re-ID methods in terms of accuracy and generalization?
• RQ5 Can SimMC be effectively applied to skeleton sequences estimated from RGB videos, and does it improve performance when fine-tuning existing models?

Key findings

• SimMC achieves a top-1 accuracy of 42.5% on the CASIA-B dataset, outperforming the SOTA method SM-SGE by 5.6 percentage points.
• On the KS20 and BIWI datasets, SimMC improves mAP by 8.6% and 0.4% respectively compared to SM-SGE, demonstrating strong generalization.
• The ablation study confirms that MPC alone boosts performance by 9.8% to 47.8% in top-1 accuracy, validating its effectiveness in mining representative features.
• Adding MIC to MPC yields an additional 0.8% to 2.5% gain in top-1 accuracy and 0.2% to 1.2% in mAP, proving that intra-sequence consistency learning enhances representation quality.
• t-SNE visualization shows that SimMC learns more separable and lower-entropy representations than AGE and SM-SGE, indicating richer semantics and better clustering.
• The framework is robust to hyperparameter changes, particularly temperature τ and mask strategy, and achieves optimal performance with appropriate fusion (λ) of MPC and MIC.