[Paper Review] Towards Coherent Visual Storytelling with Ordered Image Attention

Tom Braude, Idan Schwartz+2 more|arXiv (Cornell University)|Aug 4, 2021

Multimodal Machine Learning Applications43 references4 citations

TL;DR

This paper proposes Ordered Image Attention (OIA) for visual storytelling, modeling cross-image interactions through order-aware representation aggregation to improve coherence. By combining OIA with Image-Sentence Attention (ISA) and an adaptive prior, the method boosts METEOR by 1% on VIST and demonstrates superior story coherence, focus, and image grounding in human evaluation.

ABSTRACT

We address the problem of visual storytelling, i.e., generating a story for a given sequence of images. While each sentence of the story should describe a corresponding image, a coherent story also needs to be consistent and relate to both future and past images. To achieve this we develop ordered image attention (OIA). OIA models interactions between the sentence-corresponding image and important regions in other images of the sequence. To highlight the important objects, a message-passing-like algorithm collects representations of those objects in an order-aware manner. To generate the story's sentences, we then highlight important image attention vectors with an Image-Sentence Attention (ISA). Further, to alleviate common linguistic mistakes like repetitiveness, we introduce an adaptive prior. The obtained results improve the METEOR score on the VIST dataset by 1%. In addition, an extensive human study verifies coherency improvements and shows that OIA and ISA generated stories are more focused, shareable, and image-grounded.

Motivation & Objective

• To improve story coherence in visual storytelling by modeling dependencies between images beyond immediate sentence-image alignment.
• To address linguistic issues like repetitiveness through an adaptive prior mechanism.
• To enhance attention mechanisms by focusing on salient image regions across the sequence in an order-aware manner.
• To generate more image-grounded and shareable stories through improved attention and linguistic modeling.

Proposed method

• OIA uses a message-passing-like algorithm to collect and propagate representations of important objects across images in a sequence, preserving temporal order.
• Image-Sentence Attention (ISA) highlights relevant image regions for each sentence, ensuring alignment with both current and contextually relevant images.
• An adaptive prior is introduced to reduce repetitive language by dynamically adjusting attention to less frequent or more informative tokens.
• The model integrates OIA and ISA into a sequence-to-sequence generation framework, enabling context-aware story generation.
• The architecture is trained end-to-end on the VIST dataset with automatic metrics and human evaluation for coherence and grounding.

Experimental results

Research questions

• RQ1 Can modeling cross-image dependencies improve story coherence in visual storytelling?
• RQ2 How does order-aware representation aggregation affect the quality and focus of generated stories?
• RQ3 To what extent does an adaptive prior reduce linguistic repetition in story generation?
• RQ4 How do OIA and ISA compare to standard attention mechanisms in generating image-grounded narratives?

Key findings

• The model improves the METEOR score on the VIST dataset by 1% compared to baseline methods.
• Human evaluation confirms that stories generated with OIA and ISA are more coherent, focused, and image-grounded.
• Stories produced with the proposed method are rated more shareable than those from baseline models.
• The adaptive prior effectively reduces repetitive language, contributing to more diverse and natural-sounding narratives.
• OIA enables better modeling of long-range dependencies between images, enhancing narrative consistency.