[Paper Review] Learning Semantic Person Image Generation by Region-Adaptive Normalization

Zhengyao Lv, Xiaoming Li+5 more|arXiv (Cornell University)|Apr 14, 2021

Generative Adversarial Networks and Image Synthesis44 references4 citations

TL;DR

This paper proposes SPGNet, a two-stage framework for semantic person image generation that first predicts target semantic parsing maps and then uses region-adaptive normalization to transfer per-region appearance styles, significantly improving photo-realism and consistency over state-of-the-art methods, with a 12.613 FID score on DeepFashion and 8.06% higher Jab score than the second-best method.

ABSTRACT

Human pose transfer has received great attention due to its wide applications, yet is still a challenging task that is not well solved. Recent works have achieved great success to transfer the person image from the source to the target pose. However, most of them cannot well capture the semantic appearance, resulting in inconsistent and less realistic textures on the reconstructed results. To address this issue, we propose a new two-stage framework to handle the pose and appearance translation. In the first stage, we predict the target semantic parsing maps to eliminate the difficulties of pose transfer and further benefit the latter translation of per-region appearance style. In the second one, with the predicted target semantic maps, we suggest a new person image generation method by incorporating the region-adaptive normalization, in which it takes the per-region styles to guide the target appearance generation. Extensive experiments show that our proposed SPGNet can generate more semantic, consistent, and photo-realistic results and perform favorably against the state of the art methods in terms of quantitative and qualitative evaluation. The source code and model are available at https://github.com/cszy98/SPGNet.git.

Motivation & Objective

• To address the challenge of generating photo-realistic, semantically consistent person images under novel poses.
• To overcome limitations in existing methods that fail to preserve fine-grained appearance details and suffer from blurry or inconsistent textures.
• To decompose the complex pose transfer task into two stages: semantic map prediction followed by region-specific appearance translation.
• To improve appearance consistency and realism by leveraging per-region style guidance through region-adaptive normalization.
• To demonstrate superior performance over state-of-the-art methods in both quantitative metrics and qualitative results.

Proposed method

• A two-stage framework is proposed: first, a SPATN model predicts target semantic parsing maps from the source image and target pose.
• The predicted semantic maps are used as conditional supervision to guide appearance translation in the second stage.
• A novel region-adaptive normalization layer (SEAN) is introduced, where style codes from the source image are adapted per body region and broadcast to corresponding target regions.
• The method uses distance maps of the skeleton instead of heatmaps to improve the quality of predicted semantic parsing maps.
• The SPGNet generator employs a dual-path U-Net architecture with conditional normalization to synthesize high-fidelity images.
• Conditional convolutional kernels are also explored as an alternative to normalization, but region-adaptive normalization yields better results.

Experimental results

Research questions

• RQ1 Can predicting target semantic parsing maps improve the quality and consistency of person image generation under novel poses?
• RQ2 How does region-adaptive normalization compare to global normalization (e.g., AdaIN) in preserving per-region appearance details during pose transfer?
• RQ3 Does using distance maps instead of heatmaps for skeleton representation lead to better semantic map prediction and, consequently, better image generation?
• RQ4 Can a two-stage approach—semantic map prediction followed by region-specific style translation—outperform end-to-end pose transfer methods?
• RQ5 What is the contribution of per-region style guidance in reducing visual artifacts and improving photo-realism?

Key findings

• SPGNet achieves a FID score of 12.613 on the DeepFashion dataset, outperforming the second-best method by 13.1% in FID reduction.
• The Jab score (percentage of images judged as best) is 8.06% higher than the second-best method, indicating superior perceptual quality.
• The Ours-Full variant achieves the highest SSIM (0.783) and lowest LPIPS (0.2133), confirming improved structural and perceptual fidelity.
• Ablation studies show that region-adaptive normalization significantly improves performance over global normalization (e.g., SEAN) and other normalization variants.
• Visual comparisons confirm that Ours-Full generates more realistic textures and consistent body parts, especially in occluded or complex poses.
• The use of distance maps for skeleton representation leads to more accurate semantic map predictions, which in turn enhances the final image quality.