[Paper Review] Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation

Hyojin Bahng, Seungjoo Yoo+5 more|arXiv (Cornell University)|Apr 11, 2018

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Text2Colors, a conditional GAN framework that generates multiple diverse, semantically aligned color palettes from rich text inputs (e.g., phrases or sentences) and uses them to guide high-quality image colorization. The model outperforms ground-truth palettes in human preference studies, with 56.2% of users preferring generated palettes over real ones, demonstrating effective semantic alignment and multimodal color generation.

ABSTRACT

This paper proposes a novel approach to generate multiple color palettes that reflect the semantics of input text and then colorize a given grayscale image according to the generated color palette. In contrast to existing approaches, our model can understand rich text, whether it is a single word, a phrase, or a sentence, and generate multiple possible palettes from it. For this task, we introduce our manually curated dataset called Palette-and-Text (PAT). Our proposed model called Text2Colors consists of two conditional generative adversarial networks: the text-to-palette generation networks and the palette-based colorization networks. The former captures the semantics of the text input and produce relevant color palettes. The latter colorizes a grayscale image using the generated color palette. Our evaluation results show that people preferred our generated palettes over ground truth palettes and that our model can effectively reflect the given palette when colorizing an image.

Motivation & Objective

• To enable machines to generate multiple semantically relevant color palettes from rich text inputs such as phrases or sentences, moving beyond single-word or fixed-dataset constraints.
• To address the multimodal nature of color perception by generating diverse palettes from a single text input, reflecting varying human interpretations of semantic concepts.
• To develop a unified framework that integrates text-to-palette generation with palette-guided image colorization, ensuring faithful color transfer from palette to image.
• To create a new benchmark dataset, Palette-and-Text (PAT), with 10,183 human-curated text-palette pairs to support training and evaluation of text-based color generation.

Proposed method

• The model uses two conditional GANs: a text-to-palette generator (TPN) that maps text embeddings to multiple diverse color palettes, and a palette-based colorization network (PCN) that colorizes grayscale images using the generated palette.
• The TPN employs conditioning augmentation to generate diverse palettes from the same text input, capturing the multimodal nature of color semantics.
• The PCN integrates the generated palette into skip-connection layers of a U-Net architecture, enabling precise color propagation and semantic consistency in colorization.
• A conditional adversarial loss is applied in both networks to ensure that generated palettes and colorized images align with the input text and maintain perceptual quality.
• The model is trained end-to-end on the manually curated Palette-and-Text (PAT) dataset, which contains 10,183 paired examples of multi-word text and multi-color palettes.
• Human evaluation is used to validate both palette quality and colorization performance, with users comparing generated outputs against ground truth and baseline models.

Experimental results

Research questions

• RQ1 Can a deep generative model produce multiple diverse, semantically relevant color palettes from rich text inputs such as phrases or sentences, rather than just single words?
• RQ2 How well can a model align generated color palettes with the semantic meaning of input text, especially when compared to human-curated ground-truth palettes?
• RQ3 To what extent can a palette-guided colorization network faithfully reflect the colors and semantics of a generated palette in the final image output?
• RQ4 Can the model generalize across diverse image types, including photos and patterns, while maintaining semantic consistency in colorization?
• RQ5 How does the performance of the proposed model compare to state-of-the-art colorization baselines when using only five palette colors?

Key findings

• The text-to-palette generator (TPN) achieved a fooling rate of 56.2% in user preference studies, indicating that participants preferred the generated palettes over ground-truth palettes more than half the time.
• The palette-based colorization network (PCN) produced higher-quality colorizations than a state-of-the-art baseline model, even when using only five colors from the palette instead of 313 bins of ab gamut.
• User studies showed that the PCN effectively applied diverse colors from the palette to the image, with higher scores across all five-point Likert scale questions on palette utilization and visual quality.
• The model demonstrated strong multimodal generation, producing multiple plausible palettes from the same text input, reflecting the inherent ambiguity in human color perception.
• The proposed Palette-and-Text (PAT) dataset of 10,183 curated text-palette pairs enables training and evaluation of text-guided color generation, supporting future research in semantic color recommendation.