[Paper Review] C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis

K J Joseph, Arghya Pal+2 more|arXiv (Cornell University)|Sep 20, 2018

Multimodal Machine Learning Applications19 references4 citations

TL;DR

C4Synth proposes a novel text-to-image generation framework that leverages multiple captions per image through cross-caption cycle-consistent learning, improving image quality and diversity. By using cascaded and recurrent architectures to iteratively refine image features based on complementary caption information, the method achieves state-of-the-art inception scores on CUB and Oxford-102 datasets, with zero-shot generalization to unseen classes.

ABSTRACT

Generating an image from its description is a challenging task worth solving because of its numerous practical applications ranging from image editing to virtual reality. All existing methods use one single caption to generate a plausible image. A single caption by itself, can be limited, and may not be able to capture the variety of concepts and behavior that may be present in the image. We propose two deep generative models that generate an image by making use of multiple captions describing it. This is achieved by ensuring 'Cross-Caption Cycle Consistency' between the multiple captions and the generated image(s). We report quantitative and qualitative results on the standard Caltech-UCSD Birds (CUB) and Oxford-102 Flowers datasets to validate the efficacy of the proposed approach.

Motivation & Objective

• To address the semantic limitations of single-caption text-to-image synthesis by leveraging multiple, complementary captions per image.
• To reduce the semantic gap between textual descriptions and visual content by modeling cross-caption consistency.
• To develop a scalable architecture that adapts to variable numbers of input captions, overcoming fixed-architecture limitations.
• To generate more plausible, diverse, and detailed images by iteratively refining image features using caption feedback.

Proposed method

• C4Synth employs a cycle-consistency loss across multiple captions and generated images, ensuring that caption-to-image and image-to-caption mappings preserve semantic fidelity.
• The cascaded architecture processes captions sequentially using multiple generator-discriminator pairs, each conditioned on a caption embedding and a history block.
• The recurrent variant, Recurrent-C4Synth, uses a hidden state to maintain memory of prior caption processing, enabling dynamic adaptation to variable caption counts.
• Generative adversarial networks (GANs) are used in a conditional setup to generate 256×256 pixel images from text embeddings.
• Cross-caption cycle consistency is enforced via a cyclic mapping: G₂∘F₂∘G₁∘F₁(t) ≈ t, ensuring semantic coherence across captions.
• A fine-tuned Inception network is used to compute inception scores for quantitative evaluation.

Figure 1: The figure shows two images generated by C4Synth. The corresponding captions that are used while generating images are listed on the left side. (Best viewed in color.)

Experimental results

Research questions

• RQ1 Can multiple captions describing the same image improve the quality and diversity of generated images compared to single-caption methods?
• RQ2 How can cycle-consistent learning across multiple captions be formalized and enforced in a deep generative model for text-to-image synthesis?
• RQ3 Can a recurrent architecture effectively handle variable numbers of input captions without architectural constraints?
• RQ4 To what extent does the model generalize to unseen classes in zero-shot settings?

Key findings

• Recurrent-C4Synth achieved a state-of-the-art inception score of 3.52 ± 0.15 on the Oxford-102 Flowers dataset, outperforming prior methods.
• On the CUB dataset, Recurrent-C4Synth achieved an inception score of 4.07 ± 0.13, closely matching the performance of HD-GAN, the previous SOTA.
• Both Cascaded-C4Synth and Recurrent-C4Synth outperformed four out of five baseline methods on both datasets in terms of inception score.
• The model demonstrated zero-shot generalization, generating high-quality images for classes not seen during training on both CUB and Oxford-102 datasets.
• Qualitative results show progressive image refinement over time steps in Recurrent-C4Synth, with each caption contributing to improved image details.
• The model can generate stylized variations of the same object with different poses and backgrounds by conditioning on the same captions with different noise vectors.

(a) $Legend:\{\textbf{t}_{i}\}:\text{True Captions; }\{\hat{\textbf{t}}_{i}\}:\text{Generated Captions; }\{\hat{\textbf{I}}_{i}\}:\text{Generated Images; }\textbf{I}:\text{True Image; }\newline \text{CCCL: Cross-Caption Cycle Consistency Loss;}\text{ DL: Discriminator Loss.}$