[Paper Review] Bipartite Graph Reasoning GANs for Person Image Generation

Hao Tang, Song Bai+2 more|arXiv (Cornell University)|Aug 10, 2020

Human Pose and Action Recognition16 references36 citations

TL;DR

BiGraphGAN introduces Bipartite Graph Reasoning (BGR) and Interaction-and-Aggregation (IA) blocks to model crossing long-range pose relations and jointly enhance appearance and shape for pose-guided person image generation, achieving strong quantitative and visual results on Market-1501 and DeepFashion.

ABSTRACT

We present a novel Bipartite Graph Reasoning GAN (BiGraphGAN) for the challenging person image generation task. The proposed graph generator mainly consists of two novel blocks that aim to model the pose-to-pose and pose-to-image relations, respectively. Specifically, the proposed Bipartite Graph Reasoning (BGR) block aims to reason the crossing long-range relations between the source pose and the target pose in a bipartite graph, which mitigates some challenges caused by pose deformation. Moreover, we propose a new Interaction-and-Aggregation (IA) block to effectively update and enhance the feature representation capability of both person's shape and appearance in an interactive way. Experiments on two challenging and public datasets, i.e., Market-1501 and DeepFashion, show the effectiveness of the proposed BiGraphGAN in terms of objective quantitative scores and subjective visual realness. The source code and trained models are available at https://github.com/Ha0Tang/BiGraphGAN.

Motivation & Objective

• Motivate improved modeling of pose deformation in person image generation.
• Propose a graph-based module to capture crossing long-range relations between source and target poses.
• Develop interactive blocks to jointly enhance shape and appearance representations.
• Introduce an attention-based fusion mechanism to generate final high-quality images.
• Demonstrate effectiveness on two public datasets with comprehensive quantitative and qualitative evaluations.

Proposed method

• Introduce Bipartite Graph Reasoning (BGR) blocks to model crossing long-range relations between source and target poses via Graph Convolutional Networks (GCNs).
• Project pose features into a bipartite graph space, perform cross-relational reasoning, and map back to coordinate space with residual connections.
• Introduce Interaction-and-Aggregation (IA) blocks to interactively enhance appearance and shape features and synchronize updates.
• Use an Attention-based Image Fusion (AIF) module to selectively fuse input and intermediate results for the final image.
• Train with dual discriminators (appearance and shape) using adversarial, L1, and perceptual losses.
• Represent poses as 18-channel heat maps and share shape encoders for source and target poses.

Experimental results

Research questions

• RQ1 Can crossing long-range pose relations be effectively modeled with bipartite graph reasoning to improve pose-to-pose and pose-to-image translation?
• RQ2 Does a dedicated IA module improve joint shaping and appearance representation during generation?
• RQ3 Can attention-based fusion help produce more realistic final images by selectively combining input and intermediate results?
• RQ4 How does BiGraphGAN perform against state-of-the-art methods on Market-1501 and DeepFashion in terms of standard metrics and human judgments?

Key findings

• BiGraphGAN achieves better SSIM, Mask-SSIM, and PCKh on Market-1501 compared with several state-of-the-art methods.
• BiGraphGAN achieves better SSIM and PCKh on DeepFashion compared with several state-of-the-art methods.
• The method yields higher Inception Score (IS) than the most related model PATN on both datasets.
• Ablation studies show that both BGR branches (B2A and A2B) and the non-sharing configuration improve performance over the baseline, with AIF contributing further gains.
• A user study indicates BiGraphGAN produces more photo-realistic images than competing methods.
• Qualitative results demonstrate clearer and more plausible person images than several baselines and the SOTA PATN.