[Paper Review] Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks

Qingrong Cheng, Keyu Wen+1 more|arXiv (Cornell University)|Aug 20, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes VLMGAN*, a novel text-to-image synthesis framework using a dual vision-language matching mechanism that enhances both image realism and semantic consistency by aligning generated images with both text descriptions and real images. The method achieves state-of-the-art performance on CUB and MSCOCO datasets by integrating vision-language matching into GAN-based generators, with a new metric, VLMS, validating improved alignment between text and image.

ABSTRACT

Text-to-image synthesis aims to generate a photo-realistic and semantic consistent image from a specific text description. The images synthesized by off-the-shelf models usually contain limited components compared with the corresponding image and text description, which decreases the image quality and the textual-visual consistency. To address this issue, we propose a novel Vision-Language Matching strategy for text-to-image synthesis, named VLMGAN*, which introduces a dual vision-language matching mechanism to strengthen the image quality and semantic consistency. The dual vision-language matching mechanism considers textual-visual matching between the generated image and the corresponding text description, and visual-visual consistent constraints between the synthesized image and the real image. Given a specific text description, VLMGAN* firstly encodes it into textual features and then feeds them to a dual vision-language matching-based generative model to synthesize a photo-realistic and textual semantic consistent image. Besides, the popular evaluation metrics for text-to-image synthesis are borrowed from simple image generation, which mainly evaluates the reality and diversity of the synthesized images. Therefore, we introduce a metric named Vision-Language Matching Score (VLMS) to evaluate the performance of text-to-image synthesis which can consider both the image quality and the semantic consistency between synthesized image and the description. The proposed dual multi-level vision-language matching strategy can be applied to other text-to-image synthesis methods. We implement this strategy on two popular baselines, which are marked with ${ ext{VLMGAN}_{+ ext{AttnGAN}}}$ and ${ ext{VLMGAN}_{+ ext{DFGAN}}}$. The experimental results on two widely-used datasets show that the model achieves significant improvements over other state-of-the-art methods.

Motivation & Objective

• Address the challenge of semantic inconsistency in text-to-image synthesis, where generated images often fail to match detailed text descriptions despite high realism.
• Improve image quality and textual-visual alignment by introducing a dual multi-level vision-language matching mechanism that enforces both text-to-image and image-to-image consistency.
• Develop a new evaluation metric, Vision-Language Matching Score (VLMS), to better reflect human perception of image-text alignment beyond traditional GAN metrics.
• Demonstrate the generalization and robustness of the proposed method across diverse text descriptions and attribute variations.
• Enable the dual matching strategy to be applied as a plug-in to existing text-to-image GAN baselines, enhancing their performance without architectural overhaul.

Proposed method

• Introduce a dual vision-language matching mechanism that simultaneously enforces textual-visual matching between generated images and text descriptions, and visual-visual consistency between generated and real images.
• Use pre-trained vision and text encoders to extract multi-level local features from images and text, enabling fine-grained alignment at the regional level.
• Design a generator network conditioned on text embeddings that synthesizes images while being optimized via adversarial loss and dual matching losses.
• Formulate two key loss components: Visual-Visual Matching (VVM) loss to align generated images with real images, and Visual-Language Matching (VLM) loss to align generated images with text descriptions.
• Train the model end-to-end using a combined loss function that includes adversarial loss, VVM loss, and VLM loss, with the generator minimizing all three components.
• Propose the Vision-Language Matching Score (VLMS) as a new evaluation metric, computed by comparing image-text embeddings using a pre-trained CLIP-like model, to assess both image quality and semantic consistency.

Experimental results

Research questions

• RQ1 Can a dual vision-language matching mechanism significantly improve the semantic consistency between generated images and their corresponding text descriptions?
• RQ2 Does enforcing visual-visual consistency between generated and real images further enhance image quality and realism in text-to-image synthesis?
• RQ3 Can the proposed Vision-Language Matching Score (VLMS) serve as a more reliable and human-aligned evaluation metric than traditional GAN metrics like FID or IS?
• RQ4 To what extent can the dual matching strategy be generalized and applied to existing state-of-the-art text-to-image GAN models?
• RQ5 How robust is the method when key attributes in text descriptions are modified or when generating multiple images from the same description?

Key findings

• The proposed ${ ext{VLMGAN}_{+ ext{AttnGAN}}}$ model achieves the highest VLMS score on the MSCOCO dataset, outperforming the baseline AttnGAN and other SOTA methods.
• The ${ ext{VLMGAN}_{+ ext{DFGAN}}}$ model achieves the highest VLMS score on the CUB dataset, demonstrating superior performance on fine-grained image-text alignment.
• The VLMS metric shows strong correlation with human perception: images with lower visual quality or semantic inconsistency receive significantly lower scores.
• The model maintains high consistency across multiple generations from the same text description, with attributes like bird color and beak type preserved despite variation in pose and style.
• The method generalizes well to attribute modifications—when key attributes in the text are changed, the generated images adapt accordingly, confirming robustness.
• Training loss analysis shows that the VVM and VLM losses converge steadily, with VVM loss decreasing more significantly, indicating effective visual-visual alignment learning.