[Paper Review] Recurrent Topic-Transition GAN for Visual Paragraph Generation

Xiaodan Liang, Zhiting Hu+3 more|arXiv (Cornell University)|Mar 21, 2017

Multimodal Machine Learning ApplicationsComputer Science31 references77 citations

TL;DR

RTT-GAN jointly learns a paragraph generator and multi-level discriminators to produce diverse, semantically coherent visual paragraphs, with semi-supervised training using only single-sentence captions.

ABSTRACT

A natural image usually conveys rich semantic content and can be viewed from different angles. Existing image description methods are largely restricted by small sets of biased visual paragraph annotations, and fail to cover rich underlying semantics. In this paper, we investigate a semi-supervised paragraph generative framework that is able to synthesize diverse and semantically coherent paragraph descriptions by reasoning over local semantic regions and exploiting linguistic knowledge. The proposed Recurrent Topic-Transition Generative Adversarial Network (RTT-GAN) builds an adversarial framework between a structured paragraph generator and multi-level paragraph discriminators. The paragraph generator generates sentences recurrently by incorporating region-based visual and language attention mechanisms at each step. The quality of generated paragraph sentences is assessed by multi-level adversarial discriminators from two aspects, namely, plausibility at sentence level and topic-transition coherence at paragraph level. The joint adversarial training of RTT-GAN drives the model to generate realistic paragraphs with smooth logical transition between sentence topics. Extensive quantitative experiments on image and video paragraph datasets demonstrate the effectiveness of our RTT-GAN in both supervised and semi-supervised settings. Qualitative results on telling diverse stories for an image also verify the interpretability of RTT-GAN.

Motivation & Objective

• Motivate generating rich, diverse visual paragraphs beyond single-sentence captions.
• Propose a semi-supervised framework that leverages a paragraph corpus to guide long-form descriptions.
• Develop a region-based attention generator conditioned on hierarchical context for coherent multiple sentences.
• Employ sentence-level and topic-transition discriminators to enforce plausibility and coherent topic transitions.
• Demonstrate effectiveness on image and video paragraph datasets in supervised and semi-supervised settings.

Proposed method

• Build a generator that creates sentences recurrently using region-aware visual and language attention over semantic regions detected by dense captioning.
• Use a paragraph RNN, a sentence RNN, and a word RNN with hierarchical attention to generate multi-sentence paragraphs.
• Incorporate spatial visual attention and language attention to form a topic vector for each sentence and to copy local region phrases.
• Adopt a two-discriminator setup (sentence plausibility D^s and topic-transition coherence D^r) to guide generation via Wasserstein GAN objectives.
• Employ a reconstruction (language-model) loss for supervised data and a Monte Carlo roll-out strategy to backpropagate through discrete text samples in SeqGAN fashion.
• Extend training to semi-supervised settings by learning long-paragraph construction from a standalone paragraph corpus while using single-sentence image captions for supervision.

Experimental results

Research questions

• RQ1 Can RTT-GAN generate diverse, coherent long paragraphs for images/videos by reasoning over local semantic regions?
• RQ2 Does semi-supervised training using a paragraph corpus improve paragraph generation when only single-sentence captions are available?
• RQ3 What is the impact of region-based attention and dual discriminators on paragraph quality compared to baseline image-to-paragraph methods?
• RQ4 Can personalized paragraphs be produced by manipulating the first sentence while maintaining coherence?
• RQ5 Does the approach extend effectively to video paragraph generation with motion features?

Key findings

• RTT-GAN with full discriminators outperforms baselines on image-paragraph metrics (METEOR, CIDEr, BLEU) in fully-supervised settings.
• In semi-supervised settings, RTT-GAN (Semi-) achieves competitive CIDEr and METEOR scores using only single-sentence captions for supervision.
• Adversarial discriminators significantly improve performance; removing them degrades CIDEr by up to 4.11% in semi-supervised settings.
• Region-based attention and language-attention components substantially improve metrics over non-attention variants.
• The model supports personalized paragraph generation by altering the first sentence, producing varied, topic-coherent descriptions.
• RTT-GAN extends effectively to video paragraphs, achieving superior BLEU-4, METEOR, and CIDEr on TACoS-MultiLevel.