[Paper Review] Generating Diverse and Accurate Visual Captions by Comparative Adversarial Learning

Dianqi Li, Qiuyuan Huang+3 more|arXiv (Cornell University)|Apr 3, 2018

Multimodal Machine Learning Applications42 references43 citations

TL;DR

The paper proposes a Comparative Adversarial Learning (CAL) framework that uses a comparative relevance discriminator and policy-gradient training to produce captions that are both accurate and diverse across images, outperforming baselines on diversity and matching human judgments.

ABSTRACT

We study how to generate captions that are not only accurate in describing an image but also discriminative across different images. The problem is both fundamental and interesting, as most machine-generated captions, despite phenomenal research progresses in the past several years, are expressed in a very monotonic and featureless format. While such captions are normally accurate, they often lack important characteristics in human languages - distinctiveness for each caption and diversity for different images. To address this problem, we propose a novel conditional generative adversarial network for generating diverse captions across images. Instead of estimating the quality of a caption solely on one image, the proposed comparative adversarial learning framework better assesses the quality of captions by comparing a set of captions within the image-caption joint space. By contrasting with human-written captions and image-mismatched captions, the caption generator effectively exploits the inherent characteristics of human languages, and generates more discriminative captions. We show that our proposed network is capable of producing accurate and diverse captions across images.

Motivation & Objective

• Motivate generating captions that are not only accurate but also diverse across images.
• Introduce a novel comparative adversarial learning framework for image captioning.
• Leverage a comparative relevance discriminator to guide a caption generator toward human-like diversity.
• Propose a semantic-level caption diversity metric to evaluate cross-image diversity.
• Demonstrate improved diversity and competitive accuracy against baselines on MSCOCO.

Proposed method

• Introduce a caption generator G based on an encoder-decoder architecture that can incorporate stochastic variation via random input z.
• Propose a comparative relevance discriminator D that ranks captions by comparing a candidate caption with human-written and unrelated captions within the image-caption joint space.
• Define a comparative relevance score (cr-score) using a softmax over cosine similarities between caption embeddings and image features (Equation 3).
• Train G with policy gradient using K rollout simulations to approximate intermediate rewards Q for each generated token (Equations 5–8).
• Pretrain G with MLE and pretrain D with the comparative objective before joint adversarial training, using a rollout size K and a gamma parameter for softmax scaling.

Experimental results

Research questions

• RQ1 Can comparative adversarial learning improve caption diversity across images while preserving accuracy?
• RQ2 Does incorporating human-written captions in the comparison set help the discriminator provide richer rewards to the generator?
• RQ3 How does CAL compare to binary discriminators (G-GAN) and standard MLE in terms of diversity and automatic/ human-evaluated quality?
• RQ4 Is a semantic-level diversity metric effective for capturing cross-image caption diversity?
• RQ5 What is the impact of components like sampling, noise vectors, and the comparative discriminator on diversity?

Key findings

• CAL achieves higher diversity than baselines while maintaining competitive accuracy on MSCOCO test set according to automatic metrics.
• Human evaluation shows CAL captions are preferred over MLE and G-GAN captions, with CAL yielding more human-like descriptions.
• CAL improves caption-image retrieval recall, outperforming G-GAN and MLE, indicating better discriminability and diversity.
• A semantic-level diversity metric based on caption embedding features captures cross-image diversity and shows CAL's superiority.
• Ablation studies confirm the benefit of the comparative discriminator and rollout-based rewards for diversity gains.