[Paper Review] Bringing back simplicity and lightliness into neural image captioning

Jean-Benoit Delbrouck, Stéphane Dupont|arXiv (Cornell University)|Oct 15, 2018

Multimodal Machine Learning Applications34 references4 citations

TL;DR

This paper proposes a lightweight, efficient neural image captioning model that achieves strong performance with minimal parameters and computation by simplifying key components: using small learned embeddings (d=128), a compact cGRU decoder, pooled CNN features, and a straightforward attention mechanism. Despite its simplicity, the model outperforms stronger baselines on CIDER and METEOR metrics, demonstrating that complexity is not required for high performance.

ABSTRACT

Neural Image Captioning (NIC) or neural caption generation has attracted a lot of attention over the last few years. Describing an image with a natural language has been an emerging challenge in both fields of computer vision and language processing. Therefore a lot of research has focused on driving this task forward with new creative ideas. So far, the goal has been to maximize scores on automated metric and to do so, one has to come up with a plurality of new modules and techniques. Once these add up, the models become complex and resource-hungry. In this paper, we take a small step backwards in order to study an architecture with interesting trade-off between performance and computational complexity. To do so, we tackle every component of a neural captioning model and propose one or more solution that lightens the model overall. Our ideas are inspired by two related tasks: Multimodal and Monomodal Neural Machine Translation.

Motivation & Objective

• To develop a neural image captioning model that achieves high performance with minimal computational cost and model complexity.
• To reduce reliance on large pre-trained embeddings and complex architectures common in prior work.
• To explore the performance-complexity trade-off by simplifying each component of the captioning pipeline.
• To demonstrate that a streamlined architecture can outperform more complex models on standard metrics.
• To provide a scalable, efficient baseline for future research in neural image captioning.

Proposed method

• Uses a small, jointly-learned embedding matrix with dimension d=128 instead of large pre-trained embeddings like GloVe or word2vec.
• Employs a compact cGRU (conditional Gated Recurrent Unit) decoder with reduced hidden state size to minimize parameters and computation.
• Applies a simple, effective attention mechanism that attends over pooled CNN features (from ResNet-50) rather than full feature maps.
• Uses a single projection matrix for the output vocabulary without additional normalization or complex decoding heads.
• Trains end-to-end using cross-entropy loss and fine-tunes with policy gradient reinforcement learning (PG) to improve CIDER scores.
• Scales the model by increasing embedding and hidden dimensions or replacing the attention with multi-head attention (MHA) for further performance gains.

Experimental results

Research questions

• RQ1 Can a neural image captioning model achieve strong performance with significantly fewer parameters and less computation?
• RQ2 Does using small, jointly-learned embeddings outperform large pre-trained embeddings in image captioning?
• RQ3 Can a simplified attention mechanism based on pooled features match or exceed the performance of attention over full feature maps?
• RQ4 To what extent can a lightweight architecture be scaled to improve performance without increasing complexity?
• RQ5 Does a simpler model architecture lead to better generalization or faster training compared to complex, state-of-the-art models?

Key findings

• The proposed model achieves a CIDER score of 1.071 using RL-PG fine-tuning, surpassing the baseline XE model by +63 points.
• The model improves METEOR by +63 points over the baseline XE, outperforming even a Monte-Carlo roll-out variant.
• Using only d=128 for embeddings and a cGRU with small hidden states, the model maintains strong performance with significantly reduced parameter count.
• The model scales effectively: increasing model width (cGRUx2) or replacing the attention with multi-head attention (MHA) further improves CIDER scores.
• The model outperforms a strong baseline (Renn et al.) on all metrics, including BERTScore (B4), METEOR, and CIDER, with a simpler, faster training process.
• The ablation study confirms that the combination of small embeddings, pooled features, and a compact decoder is key to performance and efficiency.