[Paper Review] ComicGAN: Text-to-Comic Generative Adversarial Network

Ben Proven-Bessel, Zilong Zhao+1 more|arXiv (Cornell University)|Sep 19, 2021

Generative Adversarial Networks and Image Synthesis23 references4 citations

TL;DR

ComicGAN proposes a text-to-comic generative adversarial network that synthesizes comic panels from textual descriptions using custom text description generation and a comic-specific image encoder. It achieves high-quality, detailed comic outputs with improved FID scores and strong alignment between text prompts and generated visuals, outperforming baseline GANs on Dilbert-style comics.

ABSTRACT

Drawing and annotating comic illustrations is a complex and difficult process. No existing machine learning algorithms have been developed to create comic illustrations based on descriptions of illustrations, or the dialogue in comics. Moreover, it is not known if a generative adversarial network (GAN) can generate original comics that correspond to the dialogue and/or descriptions. GANs are successful in producing photo-realistic images, but this technology does not necessarily translate to generation of flawless comics. What is more, comic evaluation is a prominent challenge as common metrics such as Inception Score will not perform comparably, as they are designed to work on photos. In this paper: 1. We implement ComicGAN, a novel text-to-comic pipeline based on a text-to-image GAN that synthesizes comics according to text descriptions. 2. We describe an in-depth empirical study of the technical difficulties of comic generation using GAN's. ComicGAN has two novel features: (i) text description creation from labels via permutation and augmentation, and (ii) custom image encoding with Convolutional Neural Networks. We extensively evaluate the proposed ComicGAN in two scenarios, namely image generation from descriptions, and image generation from dialogue. Our results on 1000 Dilbert comic panels and 6000 descriptions show synthetic comic panels from text inputs resemble original Dilbert panels. Novel methods for text description creation and custom image encoding brought improvements to Frechet Inception Distance, detail, and overall image quality over baseline algorithms. Generating illustrations from descriptions provided clear comics including characters and colours that were specified in the descriptions.

Motivation & Objective

• To develop a text-to-comic generation pipeline using GANs, as no prior work exists in this domain.
• To address the challenge of generating coherent, detailed comics from textual descriptions, which differ significantly from photo-realistic image generation.
• To overcome limitations in existing text-to-image GANs by introducing domain-specific modifications for comics.
• To create a scalable method for generating diverse, descriptive text inputs for training comic GANs.
• To design and evaluate a custom image encoder tailored for comic visual features, improving generation quality over generic models.

Proposed method

• Adapted AttnGAN as the base text-to-image GAN architecture for comic generation.
• Implemented automatic text description creation via label permutation and augmentation to generate diverse, descriptive captions from comic panel labels.
• Designed a custom convolutional neural network (CNN) image encoder optimized for comic-specific visual features such as stylized lines, flat colors, and character consistency.
• Used the Frechet Inception Distance (FID) and Inception Score for quantitative evaluation, with FID serving as the primary metric due to its suitability for non-photorealistic images.
• Trained the model on 1,000 Dilbert comic panels with corresponding dialogue and 6,000 descriptive captions to enable zero-shot and in-context text-to-comic synthesis.
• Employed a modified DAMSM (Deep Visual-Semantic Hashing) loss to align generated image features with input text embeddings, improving semantic consistency.

Experimental results

Research questions

• RQ1 What unique visual and textual features in comics pose specific challenges for GAN-based image generation and evaluation?
• RQ2 How can text-to-image GANs be adapted to generate high-quality, coherent comic illustrations from natural language descriptions?
• RQ3 Can automatic text description generation from panel labels improve training data diversity and model performance in comic synthesis?
• RQ4 How does a custom comic-optimized image encoder compare to generic pre-trained models like Inception v3 in generating accurate and detailed comic panels?
• RQ5 To what extent can FID and other metrics evaluate the quality and alignment of generated comics, given the limitations of standard metrics like Inception Score?

Key findings

• ComicGAN achieved a significant improvement in FID score over baseline DCGAN and standard AttnGAN models, indicating higher visual fidelity and distributional similarity to real comic panels.
• The custom comic-specific CNN image encoder outperformed Inception v3 in feature extraction for comics, suggesting domain-specific architectures are essential for high-quality generation.
• Generated comic panels showed strong semantic alignment with input descriptions, including correct character representation and specified colors, though occasional misinterpretations of character identity occurred.
• Automatic text description generation via label permutation and augmentation successfully created diverse, descriptive captions, enabling effective training of the GAN on limited real-world data.
• The model successfully synthesized coherent, detailed comic panels from both dialogue and descriptive text inputs, demonstrating feasibility of text-to-comic generation in the Dilbert style.
• Despite the lack of annotated comic datasets, the pipeline produced results that were visually indistinguishable from original Dilbert panels in qualitative evaluation.