[Paper Review] CNN+CNN: Convolutional Decoders for Image Captioning

Qingzhong Wang, Antoni B. Chan|arXiv (Cornell University)|May 23, 2018

Multimodal Machine Learning Applications34 references70 citations

TL;DR

The paper proposes a CNN+CNN framework that replaces RNN decoders with convolutional decoders for image captioning, achieving competitive BLEU/METEOR/CIDEr scores and faster training, especially with hierarchical attention.

ABSTRACT

Image captioning is a challenging task that combines the field of computer vision and natural language processing. A variety of approaches have been proposed to achieve the goal of automatically describing an image, and recurrent neural network (RNN) or long-short term memory (LSTM) based models dominate this field. However, RNNs or LSTMs cannot be calculated in parallel and ignore the underlying hierarchical structure of a sentence. In this paper, we propose a framework that only employs convolutional neural networks (CNNs) to generate captions. Owing to parallel computing, our basic model is around 3 times faster than NIC (an LSTM-based model) during training time, while also providing better results. We conduct extensive experiments on MSCOCO and investigate the influence of the model width and depth. Compared with LSTM-based models that apply similar attention mechanisms, our proposed models achieves comparable scores of BLEU-1,2,3,4 and METEOR, and higher scores of CIDEr. We also test our model on the paragraph annotation dataset, and get higher CIDEr score compared with hierarchical LSTMs

Motivation & Objective

• Motivate replacing RNN-based decoders with CNN-based decoders for image captioning to enable parallel computation.
• Propose a CNN+CNN architecture connecting a vision CNN to a language CNN with an attention mechanism.
• Introduce a hierarchical attention module to improve alignment between image regions and language concepts.
• Systematically study the impact of language CNN width, depth, and attention on caption quality.
• Demonstrate performance on MSCOCO, Flickr30k, and a paragraph annotation dataset (PAD) and compare with state-of-the-art methods.

Proposed method

• Use a vision module based on VGG-16 (without fully connected layers) to extract image features.
• Represent sentences with a language CNN that uses causal, gated convolutional layers (GLUs) to model context.
• Incorporate an attention module via dot-product attention between image features and language concepts to form attended features.
• Employ a prediction module that fuses attention features and language concepts to predict the next word with a softmax output.
• Optionally apply hierarchical attention that feeds attention maps from lower to higher language CNN levels to refine guidance.
• Train with cross-entropy loss plus L2 regularization; inference is feed-forward using greedy word selection.

Experimental results

Research questions

• RQ1 Can a CNN-based decoder match or surpass RNN/LSTM-based decoders in image captioning performance?
• RQ2 Does hierarchical attention improve the alignment between visual regions and language concepts, boosting caption quality?
• RQ3 How do language CNN width, depth, and kernel size affect caption generation performance and required receptive field?
• RQ4 Is the CNN+CNN framework faster to train than NIC while maintaining competitive metrics?
• RQ5 How does the approach fare on paragraph-level captioning compared to sentence-level datasets?

Key findings

• CNN+CNN with and without hierarchical attention achieves competitive BLEU, METEOR, ROUGE-L, and CIDEr scores on MSCOCO and Flickr30k, with CIDEr improvements using hier-att.
• Training speed is enhanced: 6-layer CNN without hier-att trains about 3x faster than NIC on identical hardware.
• Hierarchical attention consistently improves scores compared to non-hierarchical models, notably CIDEr and METEOR on MSCOCO and Flickr30k.
• Kernel width and depth experiments show that, for MSCOCO/Flickr30k, a 6-layer network with kernel size 3 and hier-att is effective, while increasing kernel size benefits non-hierarchical setups.
• On the PAD dataset, longer captions benefit from kernel width adjustments, with kernel size 7 yielding CIDEr gains.
• Visual attention maps demonstrate meaningful focus on image regions corresponding to predicted words and reveal trade-offs in multi-object scenes.