[Paper Review] Attribute-Centric Compositional Text-to-Image Generation

Yuren Cong, Martin Renqiang Min+3 more|arXiv (Cornell University)|Jan 4, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes ACTIG, a novel attribute-centric framework for compositional text-to-image generation that improves generalization on underrepresented attribute compositions through attribute-centric feature augmentation and an image-free training paradigm. By introducing an attribute-centric contrastive loss, ACTIG reduces overfitting to overrepresented attributes and achieves state-of-the-art performance in image quality and text-image consistency on CelebA-HQ and CUB.

ABSTRACT

Despite the recent impressive breakthroughs in text-to-image generation, generative models have difficulty in capturing the data distribution of underrepresented attribute compositions while over-memorizing overrepresented attribute compositions, which raises public concerns about their robustness and fairness. To tackle this challenge, we propose ACTIG, an attribute-centric compositional text-to-image generation framework. We present an attribute-centric feature augmentation and a novel image-free training scheme, which greatly improves model's ability to generate images with underrepresented attributes. We further propose an attribute-centric contrastive loss to avoid overfitting to overrepresented attribute compositions. We validate our framework on the CelebA-HQ and CUB datasets. Extensive experiments show that the compositional generalization of ACTIG is outstanding, and our framework outperforms previous works in terms of image quality and text-image consistency.

Motivation & Objective

• Address the challenge of poor generalization in text-to-image models for underrepresented attribute compositions.
• Reduce overfitting to overrepresented attribute compositions in training data.
• Improve model robustness and fairness by learning more balanced attribute feature distributions.
• Enable high-fidelity image generation conditioned on rare or unseen attribute combinations in text prompts.
• Develop a training paradigm that does not rely on real images for underrepresented compositions.

Proposed method

• Introduce attribute-centric feature augmentation by generating text features via CLIP encoder and mapping them to image features through a text-to-image mapping network.
• Propose an image-free training scheme that trains the model using only text features and synthetic image features, bypassing real image data for rare compositions.
• Design an attribute-centric contrastive loss to disentangle attribute representations and prevent over-memorization of frequent attribute combinations.
• Alternate between fully supervised training and image-free training to balance learning from real data and rare compositions.
• Use CLIP-based text encoders and a learned mapping network to generate image features from augmented text features.
• Apply attribute extraction via string matching (CelebA-HQ) and dependency parsing (CUB) for contrastive loss training.

Experimental results

Research questions

• RQ1 Can an attribute-centric training paradigm improve generalization to underrepresented attribute compositions in text-to-image generation?
• RQ2 How effective is image-free training using synthetic image features in compensating for data sparsity in rare attribute combinations?
• RQ3 To what extent does an attribute-centric contrastive loss reduce overfitting to overrepresented attribute compositions?
• RQ4 Can the proposed framework maintain high image quality while improving text-image alignment for rare attribute combinations?
• RQ5 How does ACTIG compare to existing methods in terms of robustness, fairness, and compositional generalization?

Key findings

• ACTIG achieves state-of-the-art performance in image quality and text-image consistency on both CelebA-HQ and CUB datasets.
• The model generates high-fidelity images even for attribute compositions not seen during training, demonstrating strong compositional generalization.
• User studies show ACTIG outperforms prior models like StyleT2I and Lafite in both image quality and alignment with input text.
• The image-free training scheme effectively improves generation of underrepresented attribute compositions without relying on real images.
• The attribute-centric contrastive loss successfully reduces overfitting to frequent attribute combinations, improving model fairness.
• Qualitative results confirm that ACTIG generates more realistic and semantically aligned images than existing methods, especially for rare attribute combinations.