[Paper Review] ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation

Han Zhang, Weichong Yin+8 more|arXiv (Cornell University)|Dec 31, 2021

Multimodal Machine Learning Applications30 citations

TL;DR

ERNIE-ViLG presents a unified 10B-parameter transformer that autoregressively models both image-to-text and text-to-image generation using discrete image representations, with an end-to-end training option for text-to-image synthesis and large-scale Chinese image-text pre-training.

ABSTRACT

Conventional methods for the image-text generation tasks mainly tackle the naturally bidirectional generation tasks separately, focusing on designing task-specific frameworks to improve the quality and fidelity of the generated samples. Recently, Vision-Language Pre-training models have greatly improved the performance of the image-to-text generation tasks, but large-scale pre-training models for text-to-image synthesis task are still under-developed. In this paper, we propose ERNIE-ViLG, a unified generative pre-training framework for bidirectional image-text generation with transformer model. Based on the image quantization models, we formulate both image generation and text generation as autoregressive generative tasks conditioned on the text/image input. The bidirectional image-text generative modeling eases the semantic alignments across vision and language. For the text-to-image generation process, we further propose an end-to-end training method to jointly learn the visual sequence generator and the image reconstructor. To explore the landscape of large-scale pre-training for bidirectional text-image generation, we train a 10-billion parameter ERNIE-ViLG model on a large-scale dataset of 145 million (Chinese) image-text pairs which achieves state-of-the-art performance for both text-to-image and image-to-text tasks, obtaining an FID of 7.9 on MS-COCO for text-to-image synthesis and best results on COCO-CN and AIC-ICC for image captioning.

Motivation & Objective

• Motivate and enable bidirectional vision-language generation within a single autoregressive Transformer framework.
• Unify image-to-text and text-to-image generation using discrete image representations learned by VQVAE/VQGAN-based tokenization.
• Propose an end-to-end training method for text-to-image synthesis to jointly optimize generator and reconstructor.
• Scale pre-training to 10B parameters on a large-scale Chinese image-text dataset to achieve SOTA results in both generation directions and generative VQA.
• Demonstrate strong cross-modal semantic alignment evidenced by qualitative and quantitative experiments.

Proposed method

• Represent images as discrete sequences via VQVAE/VQGAN-based tokenization and pair them with text tokens as input to a shared transformer.
• Formulate image-to-text and text-to-image generation as autoregressive tasks within the same model using specific self-attention masks.
• For text-to-image, introduce an end-to-end training path that maps transformer hidden embeddings to image tokens and backpropagates through the image reconstructor.
• Utilize sparse attention (row/column/convolutional) to manage long visual sequences (n ~ 1024) efficiently during training and inference.
• Pre-train a 10B-parameter model on a 145M Chinese image-text pair dataset collected from Chinese webpages, image search data, and public CC/CC12M-derived captions.
• Adopt a two-stage text-to-image training for GAN-like components at scale to stabilize training, and compare against end-to-end variants on smaller scales.

Experimental results

Research questions

• RQ1 Can bidirectional image-text generation be effectively unified in a single autoregressive Transformer model?
• RQ2 Does joint pre-training on a large-scale Chinese image-text dataset yield state-of-the-art results for both text-to-image synthesis and image captioning?
• RQ3 Does an end-to-end training approach for text-to-image synthesis improve generator and reconstructor performance over the traditional two-stage pipeline?
• RQ4 How do discrete visual representations (via VQVAE/VQGAN) affect cross-modal generation quality and alignment?
• RQ5 Does the unified bidirectional model extend its semantic alignment capabilities to generative visual question answering (VQA)?

Key findings

• ERNIE-ViLG achieves state-of-the-art Fréchet Inception Distance (FID) of 7.9 on MS-COCO for text-to-image synthesis.
• In zero-shot text-to-image generation, ERNIE-ViLG attains an FID of 14.7, outperforming DALL-E (27.5).
• For image captioning, ERNIE-ViLG yields best results on COCO-CN and AIC-ICC datasets with BLEU@4, METEOR, ROUGE-L, and CIDERr metrics.
• Human evaluation shows ERNIE-ViLG outperforms CogView on image clarity, texture, and text relevance in zero-shot image generation, and yields higher caption quality on COCO-CN with fine-tuning.
• An end-to-end training approach for text-to-image synthesis improves FID by about 1.5 points over a two-stage pipeline in a lite model, and end-to-end G with end-to-end R shows the strongest gains in reconstruction quality.
• The model demonstrates strong generative VQA performance, with a Turing Test passing rate of 78.5% on FMIQA and an average quality score of 1.495.