[Paper Review] Attention-based Fusion for Multi-source Human Image Generation

Stéphane Lathuilière, Enver Sangineto+2 more|arXiv (Cornell University)|May 7, 2019

Generative Adversarial Networks and Image Synthesis34 references9 citations

TL;DR

This paper proposes an attention-based fusion mechanism for multi-source human image generation, where a person image is synthesized from a target pose and multiple source appearance images. By introducing a pose-guided feature alignment and a learnable attention module that dynamically fuses features across sources, the method achieves superior image quality and robustness, especially with variable numbers of source images, outperforming baselines on Market-1501 and DeepFashion datasets.

ABSTRACT

We present a generalization of the person-image generation task, in which a human image is generated conditioned on a target pose and a set X of source appearance images. In this way, we can exploit multiple, possibly complementary images of the same person which are usually available at training and at testing time. The solution we propose is mainly based on a local attention mechanism which selects relevant information from different source image regions, avoiding the necessity to build specific generators for each specific cardinality of X. The empirical evaluation of our method shows the practical interest of addressing the person-image generation problem in a multi-source setting.

Motivation & Objective

• To generalize the person-image generation task by conditioning on multiple source appearance images instead of a single source image.
• To address the challenge of variable input cardinality in multi-source settings, where the number of source images is not fixed a priori.
• To improve image fidelity and detail accuracy by leveraging complementary information from multiple source images, especially under occlusions or noise.
• To design a flexible, scalable architecture that avoids retraining or architectural changes for different numbers of source images.
• To validate the effectiveness of attention-based feature fusion in combining multi-source representations for high-quality image synthesis.

Proposed method

• Proposes a modified U-Net generator with an attention-based decoder that fuses features from multiple source images at each resolution level.
• Introduces a pose-guided feature encoder that deforms each source image based on the target pose, enabling spatial alignment before fusion.
• Employs a learnable attention mechanism that computes source- and location-specific weights to emphasize the most informative regions across sources.
• Uses a multi-source conditional GAN loss with a single discriminator to handle arbitrary numbers of source images, ensuring consistency and realism.
• Applies skip connections between encoder and decoder, but replaces standard feature averaging with attention-based fusion to improve feature selection.
• Introduces a 2D attention module that computes attention maps over spatial locations and source indices, enabling dynamic, context-aware fusion.

Experimental results

Research questions

• RQ1 Can multi-source appearance images improve the quality and robustness of person image generation compared to single-source methods?
• RQ2 How can a deep generative model effectively fuse features from a variable number of source images without architectural retraining?
• RQ3 What is the impact of pose-guided feature alignment on attention-based fusion performance in multi-source settings?
• RQ4 Does attention-based feature selection outperform simple averaging or concatenation in multi-source image generation?
• RQ5 How does the proposed method generalize across different numbers of source images (e.g., 2 vs. 5) in terms of image quality and detail accuracy?

Key findings

• The full model with attention-based fusion achieves the highest Fréchet Inception Distance (FID) and Inception Score (IS) on both Market-1501 and DeepFashion datasets, with IS of 3.474 on Market-1501 and 3.421 on DeepFashion when using two source images.
• The ablation study shows that removing the pose-guided feature alignment (Avg No-d) leads to a significant drop in SSIM (from 0.294 to 0.258 on Market-1501), confirming the importance of spatial alignment.
• With five source images, the full model achieves a mask-SSIM of 0.788 on Market-1501 and 0.774 on DeepFashion, outperforming the baseline Avg method by 0.026 and 0.018 respectively.
• The attention-based model (Full) reduces artifacts and improves detail generation, as shown in qualitative comparisons where it better preserves clothing details like logos and hats.
• The 2D attention variant achieves higher SSIM but lower IS than the full model, indicating a trade-off between structural fidelity and perceptual quality.
• The method demonstrates robustness to variable input cardinality, maintaining high performance across 2 and 5 source images without architectural changes.