[Paper Review] Learning Video-Story Composition via Recurrent Neural Network

Guangyu Zhong, Yi‐Hsuan Tsai+3 more|arXiv (Cornell University)|Jan 31, 2018

Video Analysis and Summarization27 references3 citations

TL;DR

This paper proposes a learning-based video-story composition framework using a two-stream recurrent neural network (RNN) that jointly models spatial-temporal semantics and motion dynamics to predict coherent clip sequences. By fusing semantic and motion RNN outputs into coherence scores and refining the order via submodular ranking to match narrative structures, the method achieves superior video-story quality over state-of-the-art approaches on benchmark datasets.

ABSTRACT

In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatial-temporal semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

Motivation & Objective

• To address the challenge of composing coherent, narratively structured video-stories from unorganized video clips.
• To model temporal coherence between video clips beyond hand-crafted features, especially in cases with ambiguous appearances or interrupted motions.
• To improve video-story quality by aligning the composition with standard storyline structures (exposition, rising action, climax, resolution).
• To develop an unsupervised learning framework that avoids reliance on expensive ground-truth annotations for video summarization or captioning.

Proposed method

• A two-stream RNN is trained to model coherence: one stream encodes spatial-temporal semantics using clip-level features, and the other captures motion dynamics via optical flow.
• The two RNNs generate separate probability scores for clip transitions, which are fused to produce a joint coherence score between video clips.
• The initial sequence is generated by greedily selecting the next clip with the highest coherence score based on previous selections.
• A submodular ranking optimization is applied to reorganize the initial sequence to better match the narrative structure, emphasizing rising dynamics and a climactic resolution.
• The framework is trained in an unsupervised manner, leveraging real video data without requiring manual story annotations.
• The final video-story is composed by solving the submodular ranking problem, ensuring smooth transitions and narrative consistency.

Experimental results

Research questions

• RQ1 Can a learning-based RNN model effectively capture temporal coherence between video clips for story composition, especially when visual appearances are ambiguous?
• RQ2 How well can joint modeling of spatial-temporal semantics and motion dynamics improve the quality of composed video-stories compared to feature-matching baselines?
• RQ3 To what extent does incorporating narrative structure (e.g., climax, resolution) via submodular ranking enhance the perceived quality and coherence of the final video-story?
• RQ4 Does the proposed method generalize to diverse video sets with varying scenes and dynamic content, such as skateboarding or walking in gardens?

Key findings

• The proposed method achieved a Bradley-Terry model score of 1.22 in user preference studies, significantly outperforming the baseline (0.88) and the Plot Analysis method (0.85).
• On the BR video set with dynamic actions like skateboarding, the submodular ranking process improved visual coherence and transition smoothness over the baseline RNN.
• The two-stream RNN baseline achieved higher AUC (area under ROC curve) than the state-of-the-art Plot Analysis method in component-wise coherence evaluation, demonstrating better learned representation of clip relationships.
• The method showed particular strength in complex scenarios with similar scenes and varying dynamics, where feature-based methods like Plot Analysis failed due to ambiguous similarity.
• Failure cases occurred primarily in uneventful or irrelevant scenes (e.g., museum visits), where content variation was low and narrative impact was weak.
• The submodular ranking process effectively enhanced results in dynamic environments by balancing motion dynamics and semantic coherence, leading to more engaging story compositions.