[Paper Review] From Captions to Visual Concepts and Back

Hao Fang, Saurabh Gupta+10 more|arXiv (Cornell University)|Nov 18, 2014

Multimodal Machine Learning Applications60 references91 citations

TL;DR

This paper proposes a novel image captioning system that trains visual detectors, language models, and a deep multimodal similarity model directly from image captions, using multiple instance learning to detect words in image regions and a maximum-entropy language model to generate fluent sentences. The system achieves state-of-the-art performance on the COCO benchmark, attaining a BLEU-4 score of 29.1% and outperforming human captions in 34% of human evaluations.

ABSTRACT

This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time.

Motivation & Objective

• To develop a caption generation system that learns visual concepts and language structure directly from image captions, avoiding reliance on separate annotated datasets for objects, attributes, and relations.
• To improve caption quality by integrating visual detectors trained via multiple instance learning on image regions, capturing salient concepts including nouns, verbs, and adjectives.
• To enhance fluency and relevance by combining a maximum-entropy language model with word detector scores and a deep multimodal similarity model for re-ranking candidate captions.
• To evaluate performance not only by automatic metrics but also via human judgment, measuring quality relative to human-written captions.

Proposed method

• Trained visual detectors for common caption words using multiple instance learning (MIL), mapping rich CNN features from image sub-regions to word labels without bounding box supervision.
• Employed a maximum-entropy language model trained on 400,000 image captions to learn word usage statistics and commonsense linguistic patterns.
• Generated candidate captions by optimizing over word detection scores to produce high-likelihood sequences that include detected words exactly once.
• Re-ranked candidate captions using a linear model with sentence-level features, including a novel Deep Multimodal Similarity Model (DMSM) that maps images and text to a shared embedding space.
• Learned re-ranking weights via Minimum Error Rate Training (MERT) using automatic metrics and human judgments on a held-out test set.
• Fine-tuned VGG and AlexNet features on the COCO dataset to improve visual representation quality.

Experimental results

Research questions

• RQ1 Can visual detectors for diverse word types—including nouns, verbs, and adjectives—be effectively trained from image captions alone, without bounding box annotations?
• RQ2 To what extent can a maximum-entropy language model trained on captions improve fluency and semantic coherence in generated captions?
• RQ3 Does incorporating a deep multimodal similarity model that aligns image and text representations enhance the selection of high-quality caption candidates?
• RQ4 Can a system trained end-to-end on captions outperform human-written captions on automatic and human evaluation metrics?

Key findings

• The system achieved a BLEU-4 score of 29.1% on the official COCO test set, surpassing human performance, which was reported at 21.7%.
• On human evaluation via Amazon Mechanical Turk, the system's captions were judged to be equal or better than human-written captions 34% of the time.
• The addition of word detector scores improved BLEU and METEOR scores, while the DMSM-based re-ranking feature led to statistically significant improvements (p < .001) in 4-gram overlap and METEOR.
• The VGG+Score+DMSM+ft variant achieved a CIDEr score of 0.925, exceeding the human CIDEr score of 0.910 on the COCO test set.
• The system was the only one to exceed human performance on 12 out of 14 official COCO metrics, including CIDEr and METEOR.
• On the PASCAL Sentence dataset, the system achieved 21.7% BLEU and 24.7% METEOR, significantly outperforming prior work such as Midge (17.6% BLEU, 19.2% METEOR).