[Paper Review] Sparsely Grouped Multi-task Generative Adversarial Networks for Facial Attribute Manipulation

Jichao Zhang, Yezhi Shu+5 more|arXiv (Cornell University)|May 19, 2018

Generative Adversarial Networks and Image Synthesis65 references3 citations

TL;DR

This paper proposes Sparsely Grouped Generative Adversarial Networks (SG-GAN), a one-input multi-output GAN architecture that enables high-quality facial attribute manipulation using only a few labeled samples per attribute group. By integrating refined residual image learning and unsupervised learning on ungrouped data, SG-GAN achieves superior performance on sparsely grouped datasets—matching or outperforming baselines even with minimal supervision—while preserving identity and background details.

ABSTRACT

Recent Image-to-Image Translation algorithms have achieved significant progress in neural style transfer and image attribute manipulation tasks. However, existing approaches require exhaustively labelling training data, which is labor demanding, difficult to scale up, and hard to migrate into new domains. To overcome such a key limitation, we propose Sparsely Grouped Generative Adversarial Networks (SG-GAN) as a novel approach that can translate images on sparsely grouped datasets where only a few samples for training are labelled. Using a novel one-input multi-output architecture, SG-GAN is well-suited for tackling sparsely grouped learning and multi-task learning. The proposed model can translate images among multiple groups using only a single commonly trained model. To experimentally validate advantages of the new model, we apply the proposed method to tackle a series of attribute manipulation tasks for facial images. Experimental results demonstrate that SG-GAN can generate image translation results of comparable quality with baselines methods on adequately labelled datasets and results of superior quality on sparsely grouped datasets. The official implementation is publicly available:https://github.com/zhangqianhui/Sparsely-Grouped-GAN.

Motivation & Objective

• To address the high annotation cost of supervised image-to-image translation by enabling learning from sparsely grouped datasets with only a few labeled samples per group.
• To overcome the limitations of existing models like StarGAN and ResidualGAN, which require fully labeled paired data or struggle with unpaired, sparse labeling.
• To develop a unified, single-model framework capable of multi-task facial attribute manipulation without requiring per-attribute model retraining.
• To stabilize adversarial training and preserve identity and background content in translated images using a novel residual learning mechanism.

Proposed method

• Proposes a one-input multi-output GAN architecture that jointly learns multiple facial attribute translations using a single shared generator and discriminator.
• Introduces a refined residual image learning (RRIL) component that replaces element-wise addition with feature concatenation to better learn the arithmetic relationship between input and residual features.
• Employs a modified reconstruction loss that excludes ungrouped (unlabeled) samples from the loss computation, enabling effective training on sparsely grouped data.
• Leverages unsupervised learning on ungrouped data to improve classification accuracy and stabilize adversarial training.
• Uses a sparsely grouped dataset setup with only a minority of samples labeled per attribute group, while the rest are used for unsupervised pre-training.
• Applies adversarial loss and identity loss to preserve content and generate high-fidelity, realistic image translations.

Experimental results

Research questions

• RQ1 Can a single GAN model achieve high-quality facial attribute manipulation across multiple attributes using only a few labeled samples per group?
• RQ2 How does the proposed refined residual image learning mechanism improve image translation quality compared to standard residual learning in low-data regimes?
• RQ3 To what extent can unsupervised learning on ungrouped data stabilize training and enhance performance in sparsely grouped learning settings?
• RQ4 How does SG-GAN compare to existing models like StarGAN and ResidualGAN in terms of visual quality and identity preservation when trained on sparse labels?
• RQ5 What is the impact of reconstruction loss design on performance when original labels are missing for some training samples?

Key findings

• SG-GAN achieves comparable or superior image translation quality to state-of-the-art models like StarGAN and ResidualGAN, even when trained on only 500 labeled samples per group.
• The refined residual image learning (RRIL) component significantly improves translation quality, increasing classification accuracy from 18.77% (no residual learning) to 93.26% for gender attribute prediction.
• SG-GAN(500) and SG-GAN(5000) achieve nearly identical performance to SG-GAN(ALL), demonstrating robustness to limited labeled data.
• The model preserves background and identity features effectively, as evidenced by high scores in identity preservation (e.g., 72 for SG-GAN(All) vs. 63 for ResidualGAN).
• The modified reconstruction loss, which excludes ungrouped samples, enables stable training on sparsely grouped data where standard StarGAN fails.
• Visual comparisons show that SG-GAN produces sharper, more realistic results than StarGAN when trained on the same sparse data, especially in preserving facial structure.