[Paper Review] Disentangled Person Image Generation

Liqian Ma, Qianru Sun+4 more|arXiv (Cornell University)|Dec 7, 2017

Video Surveillance and Tracking Methods40 references21 citations

TL;DR

This paper proposes a two-stage generative framework for disentangled person image synthesis, learning independent representations of foreground, background, and pose via a multi-branched reconstruction network followed by adversarial learning of noise-to-embedding mappings. The method enables controllable image generation and manipulation, achieving state-of-the-art performance in person re-identification using synthetic data.

ABSTRACT

Generating novel, yet realistic, images of persons is a challenging task due to the complex interplay between the different image factors, such as the foreground, background and pose information. In this work, we aim at generating such images based on a novel, two-stage reconstruction pipeline that learns a disentangled representation of the aforementioned image factors and generates novel person images at the same time. First, a multi-branched reconstruction network is proposed to disentangle and encode the three factors into embedding features, which are then combined to re-compose the input image itself. Second, three corresponding mapping functions are learned in an adversarial manner in order to map Gaussian noise to the learned embedding feature space, for each factor respectively. Using the proposed framework, we can manipulate the foreground, background and pose of the input image, and also sample new embedding features to generate such targeted manipulations, that provide more control over the generation process. Experiments on Market-1501 and Deepfashion datasets show that our model does not only generate realistic person images with new foregrounds, backgrounds and poses, but also manipulates the generated factors and interpolates the in-between states. Another set of experiments on Market-1501 shows that our model can also be beneficial for the person re-identification task.

Motivation & Objective

• To enable controllable generation of realistic person images by explicitly disentangling image factors such as foreground, background, and pose.
• To develop a two-stage framework that learns manipulatable embedding features for each disentangled factor.
• To generate high-quality, diverse person images by mapping Gaussian noise to learned embedding spaces, enabling sampling of novel combinations.
• To evaluate the disentangled representations on person re-identification, demonstrating their discriminative power.
• To construct a synthetic virtual market dataset (VM) for re-ID using disentangled generation, reducing reliance on human-annotated data.

Proposed method

• A multi-branched reconstruction network encodes input person images into disentangled embedding features for foreground, background, and pose, then reconstructs the original image from these features.
• In stage-II, three separate mapping functions are trained adversarially to map standard Gaussian noise to the learned embedding feature distributions for each factor.
• The generator is guided by pre-trained encoders and decoders from stage-I, ensuring consistency and realism in generated images.
• Adversarial training is applied to match the distribution of real embedding features (from input images) with fake features (from noise).
• The framework enables independent or joint manipulation of factors by sampling from the learned embedding spaces.
• A virtual Market-1501 dataset (VM) is constructed by fixing foreground features and varying background and pose features to generate synthetic training pairs.

Experimental results

Research questions

• RQ1 Can disentangled representations of foreground, background, and pose enable controllable person image generation?
• RQ2 Can adversarial learning of noise-to-embedding mappings produce realistic and diverse person images?
• RQ3 Can the disentangled embedding features be used to improve person re-identification performance?
• RQ4 Can synthetic image pairs generated via disentangled manipulation serve as effective training data for re-ID models?
• RQ5 How does the quality of disentangled features compare to existing unsupervised or hand-crafted feature extractors in re-ID tasks?

Key findings

• The model achieves state-of-the-art re-ID performance on the Market-1501 dataset using only the BodyROI7 feature extractor, with a rank-1 accuracy of 0.338 and mAP of 0.107.
• When using PCA to reduce feature dimensionality, the model improves to a rank-1 accuracy of 0.355, outperforming the WholeBody model and approaching supervised baselines.
• The virtual Market (VM) dataset generated via disentangled sampling achieves a rank-1 accuracy of 0.338, comparable to models trained on DukeReID with human annotations.
• With progressive unsupervised learning (PUL), the model trained on VM data reaches a rank-1 accuracy of 0.369, exceeding the performance of models trained on CUHK03 and DukeReID datasets.
• Using the KISSME metric with VM data, the model achieves a rank-1 accuracy of 0.375, demonstrating that synthetic data can match or exceed performance of real-annotated data.
• The disentangled features are not only generative but also discriminative, as evidenced by their strong performance in person re-identification tasks.