[Paper Review] CPTR: Full Transformer Network for Image Captioning

Wei Liu, Sihan Chen+3 more|arXiv (Cornell University)|Jan 26, 2021

Multimodal Machine Learning Applications20 references108 citations

TL;DR

CPTR replaces the CNN encoder with a full Transformer that sequentializes raw images into patch tokens, enabling global context modeling from the first encoder layer and achieving strong results on MSCOCO.

ABSTRACT

In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized raw images as the input to Transformer. Compared to the "CNN+Transformer" design paradigm, our model can model global context at every encoder layer from the beginning and is totally convolution-free. Extensive experiments demonstrate the effectiveness of the proposed model and we surpass the conventional "CNN+Transformer" methods on the MSCOCO dataset. Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.

Motivation & Objective

• Rethink image captioning as a sequence-to-sequence task using a full Transformer encoder.
• Eliminate convolution in the encoder by sequentially processing image patches.
• Demonstrate global context modeling at all encoder layers and analyze attention patterns.
• Show that the decoder’s words-to-patches cross-attention effectively guides caption generation.

Proposed method

• Divide the input image into fixed-size patches (e.g., 16x16) and flatten to form a patch sequence.
• Apply a linear patch embedding and learnable 1D position embeddings to feed the Transformer encoder.
• Use an encoder with stacked self-attention and feed-forward layers to model long-range dependencies from the patch sequence.
• In the decoder, employ masked self-attention and cross-attention over encoder outputs with sinusoidal word positions.
• Train with cross-entropy loss and fine-tune with self-critical training for improved captioning performance.
• Evaluate with standard metrics (BLEU, METEOR, ROUGE, CIDEr) on MSCOCO; report ablations on pretraining, image resolution, and decoder settings.

Experimental results

Research questions

• RQ1 Can a fully Transformer-based encoder (without convolution) effectively model image context for captioning by directly processing patch sequences?
• RQ2 Does processing raw image patches with self-attention enable better global context modeling than CNN-based encoders in image captioning?
• RQ3 How do patch-level self-attention and words-to-patches cross-attention influence caption quality?
• RQ4 What impact do pretraining, input resolution, and decoder configuration have on CPTR’s performance?

Key findings

• CPTR achieves higher CIDEr scores than many CNN-based and CNN+Transformer baselines on MSCOCO Karpathy test split (CIDEr 129.4).
• On online COCO test server, CPTR attains CIDEr of 129.4, outperforming several CNN+RNN and CNN+Transformer methods.
• Pretraining the encoder with ViT (ImageNet-21K) and finetuning on ImageNet 2012 yields notable CIDEr gains over training from scratch.
• Increasing input resolution from 224x224 to 384x384 with 16x16 patches substantially improves CIDEr (e.g., 116.5 when fine-tuned with pretraining).
• The model demonstrates that encoder self-attention across all layers can capture both local and global context from early layers, as visualized in attention maps.