[Paper Review] How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization

Yandong Li, Liqiang Wang+2 more|arXiv (Cornell University)|Jul 11, 2018

Video Analysis and Summarization37 references4 citations

TL;DR

This paper proposes DySeqDPP, a reinforcement learning-enhanced sequential determinantal point process that dynamically learns optimal segment lengths for local diversity in supervised video summarization. By modeling time spans adaptively and using policy gradient training, it outperforms MLE-based methods and SOTA baselines, achieving 44.3±2.8 F1 on SumMe and 58.4±2.5 on TVSum.

ABSTRACT

The large volume of video content and high viewing frequency demand automatic video summarization algorithms, of which a key property is the capability of modeling diversity. If videos are lengthy like hours-long egocentric videos, it is necessary to track the temporal structures of the videos and enforce local diversity. The local diversity refers to that the shots selected from a short time duration are diverse but visually similar shots are allowed to co-exist in the summary if they appear far apart in the video. In this paper, we propose a novel probabilistic model, built upon SeqDPP, to dynamically control the time span of a video segment upon which the local diversity is imposed. In particular, we enable SeqDPP to learn to automatically infer how local the local diversity is supposed to be from the input video. The resulting model is extremely involved to train by the hallmark maximum likelihood estimation (MLE), which further suffers from the exposure bias and non-differentiable evaluation metrics. To tackle these problems, we instead devise a reinforcement learning algorithm for training the proposed model. Extensive experiments verify the advantages of our model and the new learning algorithm over MLE-based methods.

Motivation & Objective

• Address the challenge of determining optimal local diversity span in long-form video summarization, especially for egocentric videos.
• Overcome the limitations of fixed-segment approaches in SeqDPP, which require manual partitioning and assume uniform segment lengths.
• Enable the model to automatically infer how local diversity should be enforced based on video content and temporal structure.
• Develop a training framework that mitigates exposure bias and handles non-differentiable evaluation metrics like F1-score in video summarization.
• Bridge the gap between training and inference by using reinforcement learning to align policy learning with downstream evaluation objectives.

Proposed method

• Propose DySeqDPP, a dynamic extension of SeqDPP that introduces a latent variable to learn the time span of local diversity segments from input video.
• Define local diversity via a conditional DPP over dynamically determined segments, allowing visually similar shots to co-occur if temporally separated.
• Train the model using a reinforcement learning algorithm with a policy gradient objective, where the reward is the F1-score of the generated summary.
• Use oracle summaries during training to sample segment proposals and compute shot-level scores from the diagonal of the DPP kernel matrix $\bm{L}^t$.
• Optimize the policy network via policy gradient to maximize expected F1-score, reducing exposure bias inherent in MLE-based training.
• Apply knapsack-based summary generation from scene-level scores derived from shot-level importance scores, using temporal scene splits via KTS.

Experimental results

Research questions

• RQ1 How can local diversity in video summarization be dynamically adapted to the temporal structure of long videos, rather than relying on fixed segment lengths?
• RQ2 What is the impact of learning the local diversity span end-to-end on summarization quality compared to handcrafted or fixed partitioning?
• RQ3 Can reinforcement learning effectively train a sequential DPP model with non-differentiable metrics like F1-score, and does it mitigate exposure bias?
• RQ4 How does the proposed DySeqDPP model compare to MLE-based training in terms of generalization and alignment with evaluation metrics?
• RQ5 To what extent does dynamic segment learning improve the capture of key story events in long egocentric videos?

Key findings

• DySeqDPP achieves a new state-of-the-art performance with 44.3±2.8 F1-score on the SumMe dataset, significantly outperforming the previous SOTA method SeqDPP (40.8±4.8).
• On the TVSum dataset, DySeqDPP achieves 58.4±2.5 F1-score, surpassing the previous best result of 57.4±2.0 from SeqDPP.
• Qualitative analysis shows DySeqDPP captures critical story events—such as the skydiver appearing at both start and end—that SeqDPP misses.
• The model successfully identifies key moments like a dog biting a ball in a humorous video, which SeqDPP fails to include.
• Reinforcement learning training enables better alignment between training objectives and evaluation metrics, reducing exposure bias.
• The dynamic segment learning mechanism allows the model to adapt segment lengths based on event density, improving temporal coherence and story completeness.