[Paper Review] Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks

Weihua Chen, Xianzhe Xu+6 more|arXiv (Cornell University)|Mar 30, 2023

Human Pose and Action Recognition4 citations

TL;DR

SOLIDER is a semantic controllable self-supervised learning framework that leverages prior knowledge from human images to generate pseudo semantic labels, enabling the learning of general human representations rich in semantic information. By introducing a conditional network with a semantic controller, SOLIDER produces adaptable representations with tunable semantic-appearance ratios, outperforming state-of-the-art methods across six human-centric visual tasks, establishing new baselines on all evaluated benchmarks.

ABSTRACT

Human-centric visual tasks have attracted increasing research attention due to their widespread applications. In this paper, we aim to learn a general human representation from massive unlabeled human images which can benefit downstream human-centric tasks to the maximum extent. We call this method SOLIDER, a Semantic cOntrollable seLf-supervIseD lEaRning framework. Unlike the existing self-supervised learning methods, prior knowledge from human images is utilized in SOLIDER to build pseudo semantic labels and import more semantic information into the learned representation. Meanwhile, we note that different downstream tasks always require different ratios of semantic information and appearance information. For example, human parsing requires more semantic information, while person re-identification needs more appearance information for identification purpose. So a single learned representation cannot fit for all requirements. To solve this problem, SOLIDER introduces a conditional network with a semantic controller. After the model is trained, users can send values to the controller to produce representations with different ratios of semantic information, which can fit different needs of downstream tasks. Finally, SOLIDER is verified on six downstream human-centric visual tasks. It outperforms state of the arts and builds new baselines for these tasks. The code is released in https://github.com/tinyvision/SOLIDER.

Motivation & Objective

• To learn a general, generalizable human representation from massive unlabeled human images that benefits diverse downstream human-centric visual tasks.
• To address the limitation of existing self-supervised methods in capturing sufficient semantic information, especially for tasks requiring high-level understanding like parsing and attribute recognition.
• To enable dynamic adjustment of semantic and appearance information ratios in learned representations to suit varying downstream task requirements.
• To develop a pre-trained model that can serve as a new, stronger baseline for human-centric vision tasks.

Proposed method

• SOLIDER introduces a token-level semantic classification pretext task using pseudo semantic labels generated from prior knowledge in human images, such as body parts (e.g., upper body, shoes).
• It leverages a vision transformer backbone with a learnable projection head to map features into a semantic space, supervised by the pseudo labels to enhance semantic awareness.
• A semantic controller is integrated into a conditional network architecture, allowing users to input a scalar value to control the ratio of semantic to appearance information in the final representation.
• The controller modulates the feature space during inference, enabling on-the-fly adaptation of representation composition without retraining.
• The framework is pre-trained on the LUPerson dataset (4.18M images), using contrastive learning with masked autoencoding and the new semantic pretext task.
• The final representation is fine-tuned on downstream tasks using standard head architectures, with performance evaluated across six human-centric tasks.

Figure 1 : A representation space learned by DINO [ 6 ] . Seven human images are represented in seven different colors. Each image is split into four parts according to their semantic regions, i.e . , upper body (as $\blacktriangle$ ), lower body (as + ), shoes (as $\star$ ) and background (as $\tim

Experimental results

Research questions

• RQ1 Can prior knowledge from human images be effectively leveraged to generate pseudo semantic labels for self-supervised representation learning?
• RQ2 Can a self-supervised framework learn representations with stronger semantic content compared to standard contrastive or masked autoencoding methods?
• RQ3 Can a single pre-trained model produce representations with adjustable semantic-appearance ratios suitable for diverse downstream human-centric tasks?
• RQ4 Does the proposed semantic controller enable better performance across multiple human-centric vision tasks compared to fixed-representation baselines?

Key findings

• SOLIDER achieves new state-of-the-art performance on six human-centric visual tasks, including person re-identification, attribute recognition, person search, pedestrian detection, human parsing, and pose estimation.
• On the PETA zero-shot attribute recognition benchmark, SOLIDER with Swin-B achieves 76.43% accuracy, outperforming the previous SOTA by 2.12 percentage points.
• For person search on PRW, SOLIDER achieves 59.8% mAP, a 12% relative improvement over prior SOTA methods.
• In pedestrian detection on Cityscapes, SOLIDER with Swin-S achieves 9.7% MR-2 and 39.4% H-O, outperforming previous methods under similar settings.
• On human parsing (LIP), SOLIDER with Swin-S achieves 60.21% mIOU, a 2.69% improvement over the previous SOTA.
• The average performance across five non-re-ID tasks improves from 72.6 (DINO-LUP1M) to 74.9 with SOLIDER, demonstrating the consistent benefit of enhanced semantic learning.

Figure 2 : The pipeline of the proposed SOLIDER.