[Paper Review] Towards Diverse and Natural Image Descriptions via a Conditional GAN

Bo Dai, Sanja Fidler+2 more|arXiv (Cornell University)|Mar 17, 2017

Multimodal Machine Learning ApplicationsComputer Science31 references108 citations

TL;DR

The paper introduces a conditional GAN framework (generator and evaluator) to produce diverse, natural, and visually faithful image descriptions, using policy gradient and Monte Carlo rollouts for training, and demonstrates alignment with human judgments better than MLE-based methods.

ABSTRACT

Despite the substantial progress in recent years, the image captioning techniques are still far from being perfect.Sentences produced by existing methods, e.g. those based on RNNs, are often overly rigid and lacking in variability. This issue is related to a learning principle widely used in practice, that is, to maximize the likelihood of training samples. This principle encourages high resemblance to the "ground-truth" captions while suppressing other reasonable descriptions. Conventional evaluation metrics, e.g. BLEU and METEOR, also favor such restrictive methods. In this paper, we explore an alternative approach, with the aim to improve the naturalness and diversity -- two essential properties of human expression. Specifically, we propose a new framework based on Conditional Generative Adversarial Networks (CGAN), which jointly learns a generator to produce descriptions conditioned on images and an evaluator to assess how well a description fits the visual content. It is noteworthy that training a sequence generator is nontrivial. We overcome the difficulty by Policy Gradient, a strategy stemming from Reinforcement Learning, which allows the generator to receive early feedback along the way. We tested our method on two large datasets, where it performed competitively against real people in our user study and outperformed other methods on various tasks.

Motivation & Objective

• Aim to move beyond fidelity-only captioning by promoting naturalness and diversity in image descriptions.
• Learn a generator that produces varied, human-like sentences conditioned on images.
• Train an evaluator to distinguish natural/descriptive sentences from artificial ones to guide generation.
• Provide a framework capable of generating both single sentences and descriptive paragraphs.
• Offer an evaluator that correlates more closely with human judgments than traditional metrics.

Proposed method

• Use a generator G conditioned on image features f(I) and a random vector z, implemented as an LSTM decoder.
• Use an evaluator E that embeds image and sentence representations and scores descriptions via a dot product followed by a sigmoid.
• Formulate a minimax objective L(G,E) to optimize semantic relevance and naturalness.
• Train G with Policy Gradient using an estimated expected future reward V to provide early feedback via Monte Carlo rollouts.
• Extend to paragraph generation with a hierarchical LSTM and corresponding hierarchical evaluator.
• Pretrain with MLE for G and supervised objectives for E, then jointly train G and E in an adversarial loop.
• For evaluation, beam search uses expected rewards from E rather than likelihoods.

Experimental results

Research questions

• RQ1 Can a conditional GAN framework generate image descriptions that are more natural and diverse than MLE-trained baselines?
• RQ2 Does an adversarial evaluator align more closely with human judgments than traditional n-gram based metrics?
• RQ3 How does the proposed method perform on COCO and Flickr30k datasets compared to MLE-based captioning?
• RQ4 Does the framework extend effectively to paragraph-level descriptions via a hierarchical approach?

Key findings

• G-MLE achieves higher scores than humans on several conventional metrics, highlighting metric limitations.
• G-GAN often yields more natural and diverse descriptions than G-MLE, as supported by user studies and qualitative results.
• E-GAN and E-NGAN evaluators correlate better with human judgments than BLEU/CIDEr/SPICE alone.
• Retrieval experiments show G-GAN-generated descriptions better discriminate and retrieve the correct image (higher top-k recalls).
• The evaluator trained with the GAN framework is more consistent with human evaluation than traditional metrics.
• Paragraph generation with the hierarchical extension maintains diversity and naturalness similar to single-sentence results.