[Paper Review] Video Summarisation by Classification with Deep Reinforcement Learning

Kaiyang Zhou, Tao Xiang+1 more|arXiv (Cornell University)|Jul 9, 2018

Video Analysis and Summarization9 citations

TL;DR

This paper proposes a weakly supervised video summarization method using deep reinforcement learning (DQSN) that leverages video-level category labels to guide frame selection. By combining a global recognisability reward from a pre-trained classification network with a novel dense ranking-based reward, the method effectively addresses sparse and delayed rewards in long sequences, achieving state-of-the-art performance on TVSum and CoSum datasets with faster inference than prior weakly supervised methods.

ABSTRACT

Most existing video summarisation methods are based on either supervised or unsupervised learning. In this paper, we propose a reinforcement learning-based weakly supervised method that exploits easy-to-obtain, video-level category labels and encourages summaries to contain category-related information and maintain category recognisability. Specifically, We formulate video summarisation as a sequential decision-making process and train a summarisation network with deep Q-learning (DQSN). A companion classification network is also trained to provide rewards for training the DQSN. With the classification network, we develop a global recognisability reward based on the classification result. Critically, a novel dense ranking-based reward is also proposed in order to cope with the temporally delayed and sparse reward problems for long sequence reinforcement learning. Extensive experiments on two benchmark datasets show that the proposed approach achieves state-of-the-art performance.

Motivation & Objective

• To address the limitations of unsupervised and supervised video summarization, which rely on hand-crafted criteria or expensive frame-level annotations.
• To develop a scalable, content-specific summarization method that uses only easy-to-annotate video-level category labels.
• To improve reinforcement learning-based summarization by mitigating the sparse and delayed reward problem in long sequences.
• To maintain category recognisability in summaries by aligning frame selection with semantic content.
• To achieve performance competitive with supervised methods while requiring significantly less supervision.

Proposed method

• Formulates video summarization as a sequential decision-making process using deep Q-learning (DQSN), where frames are removed based on predicted future rewards.
• Employs a companion recurrent classification network trained with supervised cross-entropy loss to provide global recognisability rewards based on whether the summary can still be correctly classified to its video category.
• Introduces a novel dense ranking-based reward that measures the relative importance of each frame by assessing how much its removal affects the classification rank of the true category.
• Combines the global recognisability reward with the dense local reward to improve credit assignment and enable effective training in long sequences.
• Trains the DQSN using deep Q-learning with experience replay and target networks to stabilize learning.
• Uses a two-stage training pipeline: first pre-train the classification network on video-level labels, then fine-tune the DQSN with reinforcement learning.

Experimental results

Research questions

• RQ1 Can video summarization be effectively trained using only video-level category labels, without frame-level annotations?
• RQ2 How can the sparse and temporally delayed reward problem in long-sequence reinforcement learning be mitigated for video summarization?
• RQ3 Can a dense, frame-level reward based on classification rank changes improve summarization performance compared to global rewards alone?
• RQ4 How does the proposed method compare to supervised and unsupervised baselines in terms of performance and scalability?
• RQ5 Can the model generalize to diverse video content while preserving semantic content and temporal coherence?

Key findings

• The proposed DQSN method achieves state-of-the-art performance on both TVSum and CoSum datasets, outperforming existing unsupervised and weakly supervised methods.
• The model improves F-scores by 1.0% on TVSum and 4.3% on CoSum compared to the previous SOTA method DR-DSN.
• DQSN outperforms the weakly supervised Backprop-Grad method by 5.9% on both datasets, demonstrating the effectiveness of the RL-based framework with dense rewards.
• The ablation study confirms that the combination of global and local rewards (r^g + r^l) yields the best performance, with the local reward significantly improving frame selection accuracy.
• DQSN is more than twice as fast as Backprop-Grad, running in 1.43 seconds per video compared to 3.21 seconds, due to a single forward pass and reduced memory usage.
• Qualitative results show that DQSN effectively captures category-relevant frames (e.g., grooming scenes) and preserves temporal storylines, while avoiding irrelevant frames that other methods mistakenly select.