[Paper Review] Contrastive Learning for Image Captioning

Bo Dai, Dahua Lin|arXiv (Cornell University)|Oct 6, 2017

Multimodal Machine Learning ApplicationsComputer Science92 citations

TL;DR

This paper introduces Contrastive Learning (CL) for image captioning, using a fixed reference model to push the target model toward distinct yet high-quality captions, showing improved results on MSCOCO and InstaPIC-1.1M across multiple metrics.

ABSTRACT

Image captioning, a popular topic in computer vision, has achieved substantial progress in recent years. However, the distinctiveness of natural descriptions is often overlooked in previous work. It is closely related to the quality of captions, as distinctive captions are more likely to describe images with their unique aspects. In this work, we propose a new learning method, Contrastive Learning (CL), for image captioning. Specifically, via two constraints formulated on top of a reference model, the proposed method can encourage distinctiveness, while maintaining the overall quality of the generated captions. We tested our method on two challenging datasets, where it improves the baseline model by significant margins. We also showed in our studies that the proposed method is generic and can be used for models with various structures.

Motivation & Objective

• Motivate theImportance of distinctiveness in image captions beyond traditional accuracy metrics.
• Propose a learning framework that encourages distinctiveness while preserving overall caption quality.
• Show that a fixed reference model can provide stable guidance for improving caption uniqueness.
• Demonstrate generality by applying CL to different captioning architectures and datasets.

Proposed method

• Define a target model p_m with parameters theta and a fixed reference model p_n with parameters phi.
• Generate positive (I,c) and negative (I,c_/) pairs, where c is a ground-truth caption for image I and c_/I is a mismatched caption for I.
• Use a log-ratio G((c,I);theta,phi)=ln p_m(c|I,theta) - ln p_n(c|I,phi) and a logistic saturating function h((c,I);theta,phi)=r_nu(G((c,I);theta,phi)).
• Maximize the objective J(theta)= (1/K)(1/T_m) sum_{k=1}^K L(theta; X, Y_k, phi) where L is the log-likelihood over positive and negative pairs.
• Balance positive and negative samples by setting nu = T_n/T_m (usually nu=1) and replicate X to sample diverse Y.
• Show that CL can be composed on top of various base models (e.g., Neuraltalk2, AdaptiveAttention) and is compatible with MLE pretraining.

Experimental results

Research questions

• RQ1 Does introducing a fixed reference model with contrastive, positive/negative constraints improve caption distinctive qualities without sacrificing overall caption quality?
• RQ2 How does CL affect standard captioning metrics (BLEU, METEOR, ROUGE-L, CIDEr) across datasets and model architectures?
• RQ3 Is CL robust across different base architectures and data scales (MSCOCO, InstaPIC-1.1M)?

Key findings

• Self-retrieval studies indicate higher distinctiveness correlates with better captioning metrics.
• On MSCOCO, AdaptiveAttention + CL improves over the base AdaptiveAttention by significant margins across BLEU, METEOR, ROUGE-L, and CIDEr, achieving strong single-model results.
• On InstaPIC-1.1M, AdaptiveAttention + CL yields substantial CIDEr gains over the base, indicating improved descriptive quality.
• CL is compatible with multiple base models and generalizes across datasets, with ablation showing that full CL (positive and negative constraints) outperforms using either constraint alone or IL/GAN baselines.
• Replacing the reference periodically can yield diminishing returns once a strong target is obtained, suggesting a single strong reference suffices in many cases.