[Paper Review] STGAN: A Unified Selective Transfer Network for Arbitrary Image Attribute Editing

Ming Liu, Yukang Ding+5 more|arXiv (Cornell University)|Apr 22, 2019

Generative Adversarial Networks and Image Synthesis32 references20 citations

TL;DR

STGAN proposes a unified selective transfer network for arbitrary image attribute editing by using the difference between target and source attribute vectors as input, rather than the full target vector, to improve training stability and focus on relevant attributes. It introduces Selective Transfer Units (STUs) that adaptively modify encoder features, enhancing both attribute manipulation accuracy and image quality, outperforming state-of-the-art methods in facial attribute editing and season translation tasks.

ABSTRACT

Arbitrary attribute editing generally can be tackled by incorporating encoder-decoder and generative adversarial networks. However, the bottleneck layer in encoder-decoder usually gives rise to blurry and low quality editing result. And adding skip connections improves image quality at the cost of weakened attribute manipulation ability. Moreover, existing methods exploit target attribute vector to guide the flexible translation to desired target domain. In this work, we suggest to address these issues from selective transfer perspective. Considering that specific editing task is certainly only related to the changed attributes instead of all target attributes, our model selectively takes the difference between target and source attribute vectors as input. Furthermore, selective transfer units are incorporated with encoder-decoder to adaptively select and modify encoder feature for enhanced attribute editing. Experiments show that our method (i.e., STGAN) simultaneously improves attribute manipulation accuracy as well as perception quality, and performs favorably against state-of-the-arts in arbitrary facial attribute editing and season translation.

Motivation & Objective

• To address the trade-off between image quality and attribute manipulation ability in attribute editing.
• To reduce the negative impact of full target attribute vectors on image reconstruction and visual fidelity.
• To improve the flexibility and accuracy of arbitrary attribute editing by focusing only on changed attributes.
• To develop a unified model that handles both local and global attribute edits effectively.
• To enhance training stability and feature consistency across encoder-decoder layers.

Proposed method

• The model inputs the difference vector between target and source attributes instead of the full target vector to reduce irrelevant guidance.
• Selective Transfer Units (STUs) are introduced at each encoder-decoder layer to adaptively select and modify encoder features based on the difference vector and hidden state.
• STUs use a learnable mechanism that combines encoder features, inner state, and the difference attribute vector to produce task-specific feature modifications.
• The modified encoder features are concatenated with decoder features to preserve spatial details and improve reconstruction.
• The architecture integrates STUs into a standard encoder-decoder framework with skip connections, maintaining high perceptual quality.
• The model is trained using adversarial loss, reconstruction loss, and identity loss to ensure realistic and accurate editing.

Experimental results

Research questions

• RQ1 Can using the difference vector between source and target attributes instead of the full target vector improve image reconstruction and reduce visual artifacts?
• RQ2 Can selective feature modification via STUs enhance both attribute manipulation accuracy and image quality simultaneously?
• RQ3 How does the selective transfer mechanism compare to standard skip connections in terms of performance trade-offs?
• RQ4 Does the STU mechanism improve generalization across diverse attribute types, including global and fine-grained attributes?
• RQ5 Can the model achieve high-quality editing without paired training data for each attribute pair?

Key findings

• STGAN achieves superior attribute generation accuracy compared to AttGAN, StarGAN, and CycleGAN on the CelebA dataset, with significant gains on global attributes like 'Gender' and 'Age'.
• The use of the difference attribute vector leads to better image reconstruction and reduced visual degradation, even when no attributes are changed.
• STGAN-dst (using full target vector) performs worse than STGAN, confirming the benefit of focusing on attribute differences.
• STU variants outperform convolutional and GRU-based alternatives, with STGAN-res showing no improvement over STGAN, indicating STUs are already optimal for selective feature modulation.
• User studies show STGAN wins in over 50% of comparisons against competing methods, indicating strong perceptual quality and realism.
• Qualitative results demonstrate high-resolution (384×384) photo-realistic editing results with minimal artifacts and consistent attribute transfer.