[Paper Review] Progressive Pose Attention Transfer for Person Image Generation

Zhen Zhu, Tengteng Huang+4 more|arXiv (Cornell University)|Apr 6, 2019

Generative Adversarial Networks and Image Synthesis46 references42 citations

TL;DR

Introduces a progressive Pose-Attentional Transfer Network (PATN) with cascaded PATBs for pose transfer, achieving better appearance/shape consistency and enabling data augmentation for re-ID.

ABSTRACT

This paper proposes a new generative adversarial network for pose transfer, i.e., transferring the pose of a given person to a target pose. The generator of the network comprises a sequence of Pose-Attentional Transfer Blocks that each transfers certain regions it attends to, generating the person image progressively. Compared with those in previous works, our generated person images possess better appearance consistency and shape consistency with the input images, thus significantly more realistic-looking. The efficacy and efficiency of the proposed network are validated both qualitatively and quantitatively on Market-1501 and DeepFashion. Furthermore, the proposed architecture can generate training images for person re-identification, alleviating data insufficiency. Codes and models are available at: https://github.com/tengteng95/Pose-Transfer.git.

Motivation & Objective

• Motivate robust pose transfer for non-rigid human images under varying poses and views.
• Develop a cascaded, attention-guided generator that progressively transfers pose while preserving appearance.
• Improve shape and appearance consistency over previous pose-transfer methods.
• Evaluate on Market-1501 and DeepFashion datasets to demonstrate quality and efficiency.
• Show potential of generated images for augmenting person re-identification datasets.

Proposed method

• Encoder converts condition image and stacked condition/target pose maps into latent codes.
• A cascade of Pose-Attentional Transfer Blocks (PATBs) progressively updates image and pose codes with pose-attention masks.
• Pose attention masks M_t guide where to sample from the condition and how to place patches for the target pose (M_t = sigmoid(conv_S(F_{t-1}^S))).
• Image code update: F_t^P = M_t ⊙ conv_P(F_{t-1}^P) + F_{t-1}^P (residual).
• Pose code update: F_t^S = conv_S(F_{t-1}^S) | F_t^P (concatenation).
• Decoder generates the final image from the final image code F_T^P; pose code F_T^S is discarded.
• Two discriminators (appearance D_A and shape D_S) judge appearance consistency and pose alignment, with final score R = R^A R^S.

Experimental results

Research questions

• RQ1 Can progressive pose transfer via pose-attentional blocks yield more realistic and geometrically consistent person images than single-step methods?
• RQ2 Does the PATN improve appearance and shape consistency while remaining computationally efficient?
• RQ3 Can generated images meaningfully augment datasets for person re-identification (re-ID) tasks?
• RQ4 How does the proposed method compare to prior pose-transfer methods on standard benchmarks (Market-1501, DeepFashion) across multiple metrics?

Key findings

• Ours achieves highest or competitive PCKh (shape consistency) on both Market-1501 and DeepFashion, with strong appearance metrics.
• On Market-1501, our method records SSIM 0.311, IS 3.323, DS 0.74, PCKh 0.94; on DeepFashion, SSIM 0.976, IS 0.96, DS 0.96, PCKh 0.96 (approximate from table text).
• Our method delivers better mask-SSIM and mask-IS than several prior works, indicating improved structural and textural fidelity.
• Model size and speed are favorable: 41.36M parameters and 60.61 fps on DeepFashion, outperforming several baselines in both efficiency and quality.
• User studies show higher realism for our method, with G2R percentages (generated rated as real) of 63.47% on Market-1501 and 31.78% on DeepFashion in the reported setup.
• Ablation studies confirm that removing PATB components or discriminator enhancements degrades performance, validating the design choices.