[Paper Review] Review Networks for Caption Generation

Zhilin Yang, Ye Yuan+3 more|arXiv (Cornell University)|May 25, 2016

Multimodal Machine Learning Applications15 references84 citations

TL;DR

This paper introduces the review network, a novel extension to the encoder-decoder framework that enhances attention mechanisms by performing multiple review steps on encoder hidden states to generate thought vectors. These thought vectors capture global, abstractive representations of input, improving performance on image captioning and source code captioning tasks beyond standard attentive encoder-decoders.

ABSTRACT

We propose a novel extension of the encoder-decoder framework, called a review network. The review network is generic and can enhance any existing encoder- decoder model: in this paper, we consider RNN decoders with both CNN and RNN encoders. The review network performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a thought vector after each review step; the thought vectors are used as the input of the attention mechanism in the decoder. We show that conventional encoder-decoders are a special case of our framework. Empirically, we show that our framework improves over state-of- the-art encoder-decoder systems on the tasks of image captioning and source code captioning.

Motivation & Objective

• Address the limitation of sequential attention in standard encoder-decoders, which lack awareness of future attention states and global context.
• Integrate discriminative supervision into the generative process in an end-to-end manner to improve caption quality.
• Develop a generic architecture that enhances existing encoder-decoder models without requiring architectural overhaul.
• Enable multitask learning by using thought vectors as inputs to auxiliary prediction heads (e.g., word occurrence prediction).

Proposed method

• Apply multiple review steps with an attention mechanism to encoder hidden states, producing a sequence of thought vectors.
• Use thought vectors as input to the decoder’s attention mechanism, replacing or augmenting the context vector.
• Formulate the review process as a recurrent operation: at each step, compute a thought vector via attention over encoder states.
• Allow weight tying between the encoder and decoder in variants to reduce parameters and improve generalization.
• Integrate thought vectors into a multitask learning setup where a secondary head predicts discriminative signals (e.g., words in a caption).
• Use both CNN and RNN encoders for image and code captioning, respectively, with RNN decoders and attention-based generation.

Experimental results

Research questions

• RQ1 Can multiple review steps with attention on encoder states improve global context modeling in encoder-decoder models?
• RQ2 How does the review network’s ability to generate abstractive, compact thought vectors affect caption generation quality?
• RQ3 Can thought vectors effectively support multitask learning, such as predicting word occurrences in captions, in an end-to-end fashion?
• RQ4 Does the review network outperform standard attentive encoder-decoders on image and source code captioning tasks?
• RQ5 Is the review network a more expressive framework than conventional attentive encoder-decoders?

Key findings

• The review network achieves state-of-the-art performance on image captioning, improving over standard attentive encoder-decoders.
• On the HabeasCorpus dataset, the review network improves log-likelihood from -5.14 (attentive encoder-decoder) to -5.06, indicating better language modeling.
• The review network achieves 38.40% character savings (CS-5) on source code captioning, outperforming the best baseline (36.51%) by over 1.8 percentage points.
• Thought vectors capture global and abstractive features such as motion, relative position, quantity, and scene context, even including objects not in the final caption.
• The review network is strictly more expressive than standard attentive encoder-decoders, as the latter is a special case of the former.
• The model consistently improves performance across both image captioning and source code captioning, demonstrating broad applicability.