[Paper Review] Hierarchical Recurrent Neural Network for Video Summarization

Bin Zhao, Xuelong Li+1 more|arXiv (Cornell University)|Apr 28, 2019

Video Analysis and Summarization35 references4 citations

TL;DR

This paper proposes H-RNN, a two-layer hierarchical recurrent neural network for video summarization that models intra-subshot and inter-subshot temporal dependencies separately. By using a unidirectional LSTM for short video subshots and a bi-directional LSTM for subshot-level dependencies, H-RNN effectively captures long-range temporal structure with reduced computation, outperforming state-of-the-art methods on the Combined and VTW datasets.

ABSTRACT

Exploiting the temporal dependency among video frames or subshots is very important for the task of video summarization. Practically, RNN is good at temporal dependency modeling, and has achieved overwhelming performance in many video-based tasks, such as video captioning and classification. However, RNN is not capable enough to handle the video summarization task, since traditional RNNs, including LSTM, can only deal with short videos, while the videos in the summarization task are usually in longer duration. To address this problem, we propose a hierarchical recurrent neural network for video summarization, called H-RNN in this paper. Specifically, it has two layers, where the first layer is utilized to encode short video subshots cut from the original video, and the final hidden state of each subshot is input to the second layer for calculating its confidence to be a key subshot. Compared to traditional RNNs, H-RNN is more suitable to video summarization, since it can exploit long temporal dependency among frames, meanwhile, the computation operations are significantly lessened. The results on two popular datasets, including the Combined dataset and VTW dataset, have demonstrated that the proposed H-RNN outperforms the state-of-the-arts.

Motivation & Objective

• To address the limitation of standard RNNs in modeling long-range temporal dependencies in long videos.
• To improve video summarization by exploiting hierarchical video structure—frames within subshots and subshots within full videos.
• To reduce computational cost and information loss compared to standard RNNs applied directly to long video sequences.
• To enhance nonlinear fitting capacity for better representation learning in video summarization.
• To develop a more effective and efficient architecture than existing RNN-based video summarization models.

Proposed method

• The model uses a two-layer hierarchical RNN: the first layer is a unidirectional LSTM processing individual video subshots.
• Each subshot is evenly cut from the original video, and the final hidden state of the first-layer LSTM is extracted as a subshot-level representation.
• The second layer is a bi-directional LSTM that processes the sequence of subshot representations to model forward and backward inter-subshot dependencies.
• The final output of the bi-directional LSTM is used to predict the probability of each subshot being selected as a key subshot in the summary.
• The hierarchical design allows long-range temporal modeling with shorter effective time steps, reducing information loss and computational load.
• The method is trained end-to-end using cross-entropy loss to optimize key subshot selection.

Experimental results

Research questions

• RQ1 Can a hierarchical RNN structure better model long-range temporal dependencies in long video sequences than standard RNNs?
• RQ2 Does separating intra-subshot and inter-subshot temporal modeling improve video summarization performance?
• RQ3 Can the hierarchical design reduce computational cost while maintaining or improving accuracy compared to standard RNNs?
• RQ4 How does the bi-directional second layer compare to unidirectional or single-layer RNNs in capturing inter-subshot dependencies?
• RQ5 To what extent does the hierarchical structure enhance nonlinear fitting for video summarization tasks?

Key findings

• H-RNN outperforms state-of-the-art methods, including vsLSTM and dppLSTM, on both the Combined and VTW datasets.
• On the VTW dataset, H-RNN achieves better performance than HD-VS, which uses a combination of 2D and 3D CNNs, due to superior long-sequence modeling.
• The bi-directional LSTM in the second layer significantly improves performance over single-directional LSTMs, confirming the importance of modeling both forward and backward context.
• H-RNN achieves better results than single and bi-directional LSTMs with fixed 80-frame input lengths, demonstrating the advantage of hierarchical processing over mean pooling or uniform sampling.
• The hierarchical structure reduces information loss and computational burden while enhancing nonlinear fitting, leading to improved summarization quality.
• Exemplar summaries generated by H-RNN closely match human-annotated key subshots, indicating strong alignment with human perception of video importance.