[Paper Review] A Frustratingly Simple Approach for End-to-End Image Captioning

Ziyang Luo, Yadong Xi+2 more|arXiv (Cornell University)|Jan 30, 2022

Multimodal Machine Learning Applications12 citations

TL;DR

This paper proposes VC-GPT, a simple yet effective end-to-end image captioning framework that connects a pre-trained CLIP-ViT visual encoder with a GPT2 language decoder via a self-ensemble cross-modal fusion mechanism, eliminating the need for object detectors. The method achieves state-of-the-art or second-best performance across MSCOCO, Flickr30k, and NoCaps benchmarks by jointly leveraging single-modal and cross-modal knowledge during generation.

ABSTRACT

Image Captioning is a fundamental task to join vision and language, concerning about cross-modal understanding and text generation. Recent years witness the emerging attention on image captioning. Most of existing works follow a traditional two-stage training paradigm. Before training the captioning models, an extra object detector is utilized to recognize the objects in the image at first. However, they require sizeable datasets with fine-grained object annotation for training the object detector, which is a daunting task. In addition, the errors of the object detectors are easy to propagate to the following captioning models, degenerating models' performance. To alleviate such defects, we propose a frustratingly simple but highly effective end-to-end image captioning framework, Visual Conditioned GPT (VC-GPT), by connecting the pre-trained visual encoder (CLIP-ViT) and language decoder (GPT2). Different from the vanilla connection method that directly inserts the cross-attention modules into GPT2, we come up with a self-ensemble cross-modal fusion mechanism that comprehensively considers both the single- and cross-modal knowledge. As a result, we do not need extra object detectors for model training. Experimental results conducted on three popular image captioning benchmarks (MSCOCO, Flickr30k and NoCaps) demonstrate that our VC-GPT achieves either the best or the second-best performance across all evaluation metrics over extensive baseline systems.

Motivation & Objective

• To eliminate the dependency on object detectors in image captioning by enabling end-to-end training.
• To improve cross-modal alignment between vision and language without fine-tuning object detectors.
• To develop a lightweight yet effective fusion mechanism that preserves pre-trained knowledge while enabling cross-modal attention.
• To achieve state-of-the-art performance on standard benchmarks using only pre-trained vision and language models.

Proposed method

• Connects CLIP-ViT (visual encoder) and GPT2 (language decoder) via a self-ensemble cross-modal fusion module to enable end-to-end training.
• Introduces a self-ensemble mechanism that combines predictions from a single-modal GPT2 expert and a cross-modal fusion expert to balance unimodal and multimodal knowledge.
• Uses cross-attention modules between visual patches and language tokens, with feature maps averaged across attention heads for interpretability.
• Employs a two-stage training protocol: cross-modal pre-training followed by fine-tuning on image captioning datasets.
• Applies positional embedding interpolation when increasing image resolution during fine-tuning to maintain alignment.
• Explores freezing strategies for CLIP-ViT and GPT2 during pre-training to preserve pre-trained knowledge.

Experimental results

Research questions

• RQ1 Can a simple fusion mechanism between a pre-trained vision encoder and a language decoder achieve state-of-the-art image captioning performance without object detectors?
• RQ2 How does self-ensemble fusion of single-modal and cross-modal experts improve captioning quality compared to vanilla cross-attention insertion?
• RQ3 What is the optimal training strategy for freezing pre-trained components (CLIP-ViT and GPT2) during cross-modal pre-training?
• RQ4 How do learning rates and image resolution during fine-tuning affect the final captioning performance?

Key findings

• VC-GPT achieves the best or second-best performance across all metrics on MSCOCO, Flickr30k, and NoCaps, outperforming extensive baseline systems.
• Freezing CLIP-ViT during pre-training improves downstream performance, yielding 90.8 CIDEr, 24.3 B@4, 28.7 M, and 23.6 S on MSCOCO (val).
• Using the same learning rate (3e-5) for GPT2 and the fusion module yields the best performance: 126.7 CIDEr, 37.6 B@4, 30.0 M, and 23.6 S on MSCOCO (val).
• Increasing image resolution during fine-tuning improves performance up to a point, but larger resolutions can degrade results due to domain shift.
• Attention maps show that VC-GPT learns visual grounding, correctly attending to objects like 'boy' and 'cat' and capturing relationships such as 'ball' and 'bat'.
• The self-ensemble mechanism effectively balances unimodal and cross-modal knowledge, leading to more accurate and diverse captions.