[論文レビュー] ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation

Han Zhang, Weichong Yin+8 more|arXiv (Cornell University)|Dec 31, 2021

Multimodal Machine Learning Applications被引用数 30

ひとこと要約

ERNIE-ViLGは、離散的な画像表現を用いて画像→テキストとテキスト→画像の生成の両方を自回帰的にモデル化する統一の10Bパラメータ変換器を提示し、テキスト→画像合成のエンドツーエンド訓練オプションと大規模な中国語画像テキスト事前訓練を提供します。

ABSTRACT

Conventional methods for the image-text generation tasks mainly tackle the naturally bidirectional generation tasks separately, focusing on designing task-specific frameworks to improve the quality and fidelity of the generated samples. Recently, Vision-Language Pre-training models have greatly improved the performance of the image-to-text generation tasks, but large-scale pre-training models for text-to-image synthesis task are still under-developed. In this paper, we propose ERNIE-ViLG, a unified generative pre-training framework for bidirectional image-text generation with transformer model. Based on the image quantization models, we formulate both image generation and text generation as autoregressive generative tasks conditioned on the text/image input. The bidirectional image-text generative modeling eases the semantic alignments across vision and language. For the text-to-image generation process, we further propose an end-to-end training method to jointly learn the visual sequence generator and the image reconstructor. To explore the landscape of large-scale pre-training for bidirectional text-image generation, we train a 10-billion parameter ERNIE-ViLG model on a large-scale dataset of 145 million (Chinese) image-text pairs which achieves state-of-the-art performance for both text-to-image and image-to-text tasks, obtaining an FID of 7.9 on MS-COCO for text-to-image synthesis and best results on COCO-CN and AIC-ICC for image captioning.

研究の動機と目的

• Motivate and enable bidirectional vision-language generation within a single autoregressive Transformer framework.
• Unify image-to-text and text-to-image generation using discrete image representations learned by VQVAE/VQGAN-based tokenization.
• Propose an end-to-end training method for text-to-image synthesis to jointly optimize generator and reconstructor.
• Scale pre-training to 10B parameters on a large-scale Chinese image-text dataset to achieve SOTA results in both generation directions and generative VQA.
• Demonstrate strong cross-modal semantic alignment evidenced by qualitative and quantitative experiments.

提案手法

• Represent images as discrete sequences via VQVAE/VQGAN-based tokenization and pair them with text tokens as input to a shared transformer.
• Formulate image-to-text and text-to-image generation as autoregressive tasks within the same model using specific self-attention masks.
• For text-to-image, introduce an end-to-end training path that maps transformer hidden embeddings to image tokens and backpropagates through the image reconstructor.
• Utilize sparse attention (row/column/convolutional) to manage long visual sequences (n ~ 1024) efficiently during training and inference.
• Pre-train a 10B-parameter model on a 145M Chinese image-text pair dataset collected from Chinese webpages, image search data, and public CC/CC12M-derived captions.
• Adopt a two-stage text-to-image training for GAN-like components at scale to stabilize training, and compare against end-to-end variants on smaller scales.]
• research_questions: [
• Can bidirectional image-text generation be effectively unified in a single autoregressive Transformer model?
• Does joint pre-training on a large-scale Chinese image-text dataset yield state-of-the-art results for both text-to-image synthesis and image captioning?
• Does an end-to-end training approach for text-to-image synthesis improve generator and reconstructor performance over the traditional two-stage pipeline?
• How do discrete visual representations (via VQVAE/VQGAN) affect cross-modal generation quality and alignment?
• Does the unified bidirectional model extend its semantic alignment capabilities to generative visual question answering (VQA)?

実験結果

リサーチクエスチョン

• RQ1 Can bidirectional image-text generation be effectively unified in a single autoregressive Transformer model?
• RQ2 Does joint pre-training on a large-scale Chinese image-text dataset yield state-of-the-art results for both text-to-image synthesis and image captioning?
• RQ3 Does an end-to-end training approach for text-to-image synthesis improve generator and reconstructor performance over the traditional two-stage pipeline?
• RQ4 How do discrete visual representations (via VQVAE/VQGAN) affect cross-modal generation quality and alignment?
• RQ5 Does the unified bidirectional model extend its semantic alignment capabilities to generative visual question answering (VQA)?

主な発見

• ERNIE-ViLG は MS-COCO のテキスト対画像合成で Fréchet Inception Distance (FID) の最新記録である 7.9 を達成。
• ゼロショットのテキスト→画像生成では ERNIE-ViLG が FID 14.7 を達成し、DALL-E（27.5）を上回る。
• 画像キャプショニングでは COCO-CN および AIC-ICC データセットで BLEU@4、METEOR、ROUGE-L、CIDERr 指標で最高結果を達成。
• 人間評価では、ゼロショット画像生成で CogView より画質・質感・テキスト関連性で上回り、微調整付きの COCO-CN でより高いキャプション品質を示す。
• エンドツーエンドのテキスト→画像合成訓練は lite モデルの二段階パイプラインより FID を約 1.5 点改善し、エンドツーエンド G とエンドツーエンド R の組み合わせが再構成品質で最も大きな改善を示す。
• モデルは高い生成的 VQA 能力を示し、FMIQA でのチューリングテスト合格率は 78.5%、平均品質スコアは 1.495。