[Paper Review] GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation

Taehyeong Kim, Min-Oh Heo+3 more|arXiv (Cornell University)|May 28, 2018

Multimodal Machine Learning Applications31 references53 citations

TL;DR

GLAC Net introduces a two-level glocal attention and context cascading to generate coherent multi-image stories, achieving competitive METEOR scores on VIST without heavy beam search.

ABSTRACT

The task of multi-image cued story generation, such as visual storytelling dataset (VIST) challenge, is to compose multiple coherent sentences from a given sequence of images. The main difficulty is how to generate image-specific sentences within the context of overall images. Here we propose a deep learning network model, GLAC Net, that generates visual stories by combining global-local (glocal) attention and context cascading mechanisms. The model incorporates two levels of attention, i.e., overall encoding level and image feature level, to construct image-dependent sentences. While standard attention configuration needs a large number of parameters, the GLAC Net implements them in a very simple way via hard connections from the outputs of encoders or image features onto the sentence generators. The coherency of the generated story is further improved by conveying (cascading) the information of the previous sentence to the next sentence serially. We evaluate the performance of the GLAC Net on the visual storytelling dataset (VIST) and achieve very competitive results compared to the state-of-the-art techniques. Our code and pre-trained models are available here.

Motivation & Objective

• Motivate the task of multi-image cued story generation and stress the need for image-specific yet contextually coherent sentences.
• Propose GLAC Net with two-level (global and local) attention to ground each sentence in both overall sequence context and image-specific features.
• Incorporate a cascading mechanism to pass sentence-level context from one sentence to the next to improve story coherence.
• Achieve competitive performance on the VIST dataset and provide code and pretrained models for reproducibility.

Proposed method

• Extract image features with ResNet-152 for each image.
• Encode the image sequence with a bi-directional LSTM to capture global context.
• Compute glocal vectors by concatenating image-specific features with bi-LSTM outputs (hard attention).
• Feed glocal vectors into decoders to generate sentences for each image.
• Apply a cascading mechanism by initializing each sentence generator with the last hidden state of the previous sentence.
• Use a sampling/count-based post-processing strategy to reduce word duplication and increase diversity.

Experimental results

Research questions

• RQ1 Can a two-level attention mechanism (global context and image-specific features) improve image-grounded story generation for sequences of images?
• RQ2 Does cascading sentence-level context across the sequence enhance overall coherence of the generated story?
• RQ3 What is the impact of global/local attention, cascading, and post-processing components on standard metrics (e.g., METEOR, perplexity) on the VIST dataset?

Key findings

• GLAC Net is competitive with baselines on the VIST dataset without beam search.
• Ablations show contributions of cascading, global attention, local attention, and post-processing to performance.
• Complete GLAC Net achieves the best overall METEOR-like performance among the tested configurations (METEOR ~0.3014 with reported setup).
• Heuristics to reduce duplicates (Count) improve METEOR and reduce repetition.
• Compared to LSTM Seq2Seq baselines, GLAC Net generally yields better performance across evaluated settings.