[Paper Review] Hierarchically-Attentive RNN for Album Summarization and Storytelling

Licheng Yu, Mohit Bansal+1 more|arXiv (Cornell University)|Aug 9, 2017

Multimodal Machine Learning Applications9 citations

TL;DR

This paper proposes a hierarchically-attentive RNN model for end-to-end visual storytelling that jointly selects representative photos from full albums and generates coherent, natural language stories. The model uses three stacked bi-directional GRUs to encode album context, attend to salient photos, and generate fluent narratives, achieving state-of-the-art performance on photo selection, story generation, and album retrieval tasks, with human evaluations showing strong preference over baselines.

ABSTRACT

We address the problem of end-to-end visual storytelling. Given a photo album, our model first selects the most representative (summary) photos, and then composes a natural language story for the album. For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story. Automatic and human evaluations show our model achieves better performance on selection, generation, and retrieval than baselines.

Motivation & Objective

• To address the challenge of end-to-end visual storytelling from full photo albums, where both photo selection and story generation must be learned jointly.
• To improve upon existing methods that rely on pre-selected summary photos or weak supervision by learning representative photo selection implicitly during training.
• To develop a unified model that performs both album summarization and story generation using hierarchical attention across multiple RNN stages.
• To evaluate the model not only on story generation but also on a novel album retrieval task, where stories are used to retrieve their source albums.
• To demonstrate that the learned photo selection mechanism aligns reasonably well with human preferences, even without explicit supervision on selection.

Proposed method

• The model uses a bi-directional GRU-RNN to encode the visual features of all photos in an album, capturing both local and global context.
• A second RNN computes attention weights over the album photos to identify the most representative (summary) photos, acting as a differentiable photo selector.
• A third RNN decodes the weighted representation of selected photos into a coherent, multi-sentence story using autoregressive generation.
• The entire model is trained end-to-end using a joint loss combining cross-entropy for story generation and a ranking regularization loss to improve photo selection.
• The ranking regularization term encourages the model to assign higher attention weights to photos that are closer to human-selected summaries.
• During inference, beam search with beam size=3 is used to generate diverse and high-quality stories.

Experimental results

Research questions

• RQ1 Can a single neural network jointly learn to select representative photos and generate a coherent story from a full photo album without relying on pre-selected summaries?
• RQ2 How well does the model's learned photo selection align with human-annotated summary photos?
• RQ3 Does the proposed hierarchical attention mechanism improve story generation quality compared to standard seq2seq or attention-based baselines?
• RQ4 Can the model be used effectively for album retrieval, where a story is used to retrieve its source album?
• RQ5 Does the inclusion of ranking regularization in the loss function significantly improve the quality of photo selection and downstream tasks?

Key findings

• The h-attn-rank model achieved 45.51% R@10 and 28.77% Meteor score on album summarization, significantly outperforming the baseline enc-dec (10.70% R@10, 18.30% Meteor).
• Human evaluation showed a statistically significant preference (p=0.01) for the h-attn-rank model over both enc-dec and enc-attn-dec baselines in forced-choice comparisons.
• The model’s learned photo selection achieved a median rank of 7.0, indicating strong alignment with human-selected summaries, and outperformed DPP-based baselines in precision and recall.
• On the album retrieval task, the h-attn-rank model achieved 57.60% Recall@10, significantly outperforming the enc-dec baseline (41.40% R@10), demonstrating strong retrieval capability.
• Even when using an oracle of human-selected 5-photo sets, the performance gain over the end-to-end model was marginal, suggesting the model learns near-optimal selection.
• The ranking regularization term improved performance on summarization and retrieval, though its impact on retrieval was less pronounced, indicating room for further optimization.