[Paper Review] SiCL: Silhouette-Driven Contrastive Learning for Unsupervised Person Re-Identification with Clothes Change

Mingkun Li, Peng Xu+2 more|arXiv (Cornell University)|May 23, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes MaskCL, a novel unsupervised contrastive learning framework for long-term person re-identification under clothes change, using person silhouette masks as semantic prompts to learn cross-clothes invariant features. By constructing a hierarchical semantic neighbor structure from RGB and mask features, MaskCL significantly outperforms unsupervised baselines and closes the gap to supervised SOTA on five benchmark datasets.

ABSTRACT

In this paper, we address a highly challenging yet critical task: unsupervised long-term person re-identification with clothes change. Existing unsupervised person re-id methods are mainly designed for short-term scenarios and usually rely on RGB cues so that fail to perceive feature patterns that are independent of the clothes. To crack this bottleneck, we propose a silhouette-driven contrastive learning (SiCL) method, which is designed to learn cross-clothes invariance by integrating both the RGB cues and the silhouette information within a contrastive learning framework. To our knowledge, this is the first tailor-made framework for unsupervised long-term clothes change eid{}, with superior performance on six benchmark datasets. We conduct extensive experiments to evaluate our proposed SiCL compared to the state-of-the-art unsupervised person reid methods across all the representative datasets. Experimental results demonstrate that our proposed SiCL significantly outperforms other unsupervised re-id methods.

Motivation & Objective

• To address the lack of unsupervised long-term person re-identification methods under clothes change, a realistic but underexplored setting.
• To overcome the failure of conventional unsupervised methods that rely solely on RGB features and cluster based on clothing color, leading to error propagation.
• To learn clothes-independent, cross-clothes invariant features by integrating silhouette masks as semantic prompts in contrastive learning.
• To establish a benchmark for unsupervised long-term person re-identification with clothes change through extensive evaluation on five datasets.

Proposed method

• MaskCL employs a two-branch network to extract RGB and silhouette mask features, enabling joint learning of appearance and semantic structure.
• Silhouette masks are used as semantic prompts to guide contrastive learning, promoting invariance to clothing changes.
• A hierarchical semantic neighbor structure is constructed: instance-level clustering via RGB features and cluster-level neighbor discovery using combined RGB and mask features.
• The model trains using contrastive loss that encourages positive pairs (same identity) to be close in the feature space, with semantic neighbors from mask features guiding the learning.
• The framework uses a neighbor sampling strategy based on a semantic neighbor set to improve feature alignment across different clothing appearances.
• The method is trained end-to-end with a contrastive objective that leverages both RGB and mask-based semantic cues to minimize feature disparity due to clothes change.

Figure 1: Illustration of hierarchical semantic clustering. At a lower level, clustering is used to disclose the instance neighbor structure; at a higher level, semantic features and RGB data are combined to expose the cluster neighbor structure.

Experimental results

Research questions

• RQ1 Can unsupervised person re-identification models generalize to long-term scenarios with significant clothing changes?
• RQ2 Can silhouette masks serve as effective semantic prompts to guide contrastive learning and improve cross-clothes invariance?
• RQ3 Does a hierarchical semantic neighbor structure, combining RGB and mask features, lead to better feature learning than RGB-only clustering?
• RQ4 How does the quality of extracted silhouette masks affect model performance in low- and high-quality datasets?
• RQ5 Can an unsupervised method achieve performance close to supervised SOTA on real-world, unconstrained clothes-change benchmarks?

Key findings

• MaskCL achieves 61.7% mAP and 71.7% R-1 on VC-Clothes, outperforming the next best unsupervised method by over 2% in mAP.
• On the challenging Deepchange benchmark, MaskCL achieves 11.8% mAP and 39.7% R-1 with ResNet-50, surpassing the unsupervised SOTA SpCL (8.90% mAP).
• The model closes the performance gap to supervised SOTA, achieving 14.9% mAP on Deepchange with ViT-B16, compared to 14.9% for supervised ViT-B16.
• On VC-Clothes, using only mask features (without RGB) achieved the best result (63.4% mAP), indicating high-quality masks contain rich invariant information.
• Visualization shows MaskCL produces more precise matches than CACL, with fewer incorrect matches due to color similarity.
• The proportion of correct identity samples in semantic neighbor sets increases during training, confirming improved feature alignment and cluster consistency.

Figure 2: Visualizing the intrinsic challenges in long-term person re-id with clothes change. We randomly selected $28$ images of a single individual from the Deepchange [ 48 ] dataset. It is clear that the differences in appearance across different clothes (rows) are much more significant than that