[Paper Review] HumanGAN: A Generative Model of Humans Images

Kripasindhu Sarkar, Lingjie Liu+2 more|arXiv (Cornell University)|Mar 11, 2021

Generative Adversarial Networks and Image Synthesis46 references4 citations

TL;DR

HumanGAN is a novel generative adversarial model that enables disentangled, part-level control over human image generation by encoding pose-independent appearance in part-specific latent vectors, which are then warped to different poses. It achieves state-of-the-art performance in global appearance sampling, pose transfer, and part-level generation at 512×512 resolution with high realism and diversity.

ABSTRACT

Generative adversarial networks achieve great performance in photorealistic image synthesis in various domains, including human images. However, they usually employ latent vectors that encode the sampled outputs globally. This does not allow convenient control of semantically-relevant individual parts of the image, and is not able to draw samples that only differ in partial aspects, such as clothing style. We address these limitations and present a generative model for images of dressed humans offering control over pose, local body part appearance and garment style. This is the first method to solve various aspects of human image generation such as global appearance sampling, pose transfer, parts and garment transfer, and parts sampling jointly in a unified framework. As our model encodes part-based latent appearance vectors in a normalized pose-independent space and warps them to different poses, it preserves body and clothing appearance under varying posture. Experiments show that our flexible and general generative method outperforms task-specific baselines for pose-conditioned image generation, pose transfer and part sampling in terms of realism and output resolution.

Motivation & Objective

• To address the lack of fine-grained, disentangled control in existing GAN-based human image generation models.
• To enable independent sampling and manipulation of body parts and garment styles while preserving identity and appearance across poses.
• To unify multiple human image generation tasks—global sampling, pose transfer, part sampling, and garment transfer—into a single generative framework.
• To overcome limitations of existing methods that either lack stochasticity or cannot perform localized appearance editing.
• To achieve high-fidelity, 512×512 resolution image synthesis with persistent identity and realistic appearance under pose variation.

Proposed method

• Uses a conditional variational autoencoder (VAE) framework to model the posterior distribution of part-specific latent vectors in a pose-independent appearance space.
• Encodes appearance using part-based noise vectors ($\bm{z}[k]$) corresponding to individual body parts (e.g., head, upper body), enabling localized control.
• Applies spatial warping to the encoded part vectors using DensePose supervision to align them with a target pose before image reconstruction.
• Constructs a noise embedding from combined body and garment part vectors to enable joint generation of body and clothing with style transfer.
• Employs a high-fidelity generator network conditioned on the warped latent vectors to synthesize photorealistic images.
• Introduces a novel disentanglement strategy that decouples pose from appearance by conditioning latent vectors on body parts and warping them independently.

Experimental results

Research questions

• RQ1 Can a single generative model jointly support global appearance sampling, pose transfer, and part-level editing in human image generation?
• RQ2 How can pose-invariant appearance representations be effectively disentangled from pose-dependent transformations in a differentiable and controllable manner?
• RQ3 To what extent can part-specific latent vectors enable diverse and realistic sampling of individual body parts without affecting the rest of the image?
• RQ4 Can the model achieve state-of-the-art performance in both realism and resolution (512×512) while supporting multiple editing tasks?
• RQ5 How does the model compare to task-specific baselines in terms of identity preservation and visual quality for pose transfer and part sampling?

Key findings

• HumanGAN outperforms task-specific baselines in global appearance sampling, achieving higher realism and diversity at 512×512 resolution.
• In user studies, 65.62% of participants preferred HumanGAN for identity preservation in pose transfer, compared to 21.88% for CBI and 12.5% for NHRR.
• For realism, 81.25% of users preferred HumanGAN, significantly outperforming CBI (6.25%) and NHRR (12.5%).
• Quantitative evaluation shows a 22% improvement in Variation–Part (0.37 vs. 0.45) and a 75% reduction in Variation–Rest (0.11 vs. 0.44) compared to the NoParts baseline, confirming superior part-level disentanglement.
• The model successfully enables garment transfer by combining part-specific latent vectors from separate body and garment images, producing coherent and realistic results.
• Despite limitations such as occasional spurious part interleaving and gender bias (attributed to the DeepFashion dataset), HumanGAN demonstrates robust performance across all evaluated tasks.