[Paper Review] Adaptive Video Highlight Detection by Learning from User History

Mrigank Rochan, Mahesh Kumar Krishna Reddy+2 more|arXiv (Cornell University)|Jul 19, 2020

Video Analysis and Summarization54 references4 citations

TL;DR

This paper proposes Adaptive-H-FCSN, a user-adaptive video highlight detection framework that leverages a user’s history of previously created highlight GIFs to personalize highlight predictions. By introducing a temporal-adaptive instance normalization (T-AIN) layer that dynamically modulates the highlight detection network based on encoded user history, the model achieves significantly improved accuracy and personalization on a large-scale dataset, outperforming generic and prior adaptive models.

ABSTRACT

Recently, there is an increasing interest in highlight detection research where the goal is to create a short duration video from a longer video by extracting its interesting moments. However, most existing methods ignore the fact that the definition of video highlight is highly subjective. Different users may have different preferences of highlight for the same input video. In this paper, we propose a simple yet effective framework that learns to adapt highlight detection to a user by exploiting the user's history in the form of highlights that the user has previously created. Our framework consists of two sub-networks: a fully temporal convolutional highlight detection network $H$ that predicts highlight for an input video and a history encoder network $M$ for user history. We introduce a newly designed temporal-adaptive instance normalization (T-AIN) layer to $H$ where the two sub-networks interact with each other. T-AIN has affine parameters that are predicted from $M$ based on the user history and is responsible for the user-adaptive signal to $H$. Extensive experiments on a large-scale dataset show that our framework can make more accurate and user-specific highlight predictions.

Motivation & Objective

• To address the subjectivity of video highlights by personalizing detection to individual user preferences.
• To overcome the limitations of generic highlight detection models that ignore user-specific interests.
• To develop a method that effectively utilizes visual user history (previously created GIF highlights) for adaptive prediction.
• To design a computationally efficient framework that processes full videos without requiring shot boundary detection.
• To demonstrate transferability of the learned highlight detection model to video summarization on small datasets.

Proposed method

• The framework consists of two sub-networks: a fully temporal convolutional highlight detection network (H) and a history encoder network (M) that processes user history.
• A novel temporal-adaptive instance normalization (T-AIN) layer is introduced into H, where affine parameters (γ and δ) are predicted from M based on user history.
• The T-AIN layer enables user-specific adaptation of the highlight detection network by modulating feature normalization using history-encoded embeddings.
• The model is trained end-to-end using a large-scale dataset of user-created highlight GIFs, avoiding the need for precomputed shot boundaries.
• The history encoder M processes multiple highlight GIFs from a user’s history and produces a compact representation to condition the T-AIN layers in H.
• The framework avoids segment-level sampling and instead processes full videos, preserving long-range temporal dependencies.

Experimental results

Research questions

• RQ1 Can user history in the form of previously created highlight GIFs improve the accuracy and personalization of video highlight detection?
• RQ2 How does the proposed T-AIN layer compare to direct concatenation or early fusion of user history and video features in modeling user preferences?
• RQ3 How sensitive is the model to the size of a user’s history, and does performance improve with more historical highlights?
• RQ4 Can a highlight detection model pre-trained on a large-scale dataset of user-created highlights generalize effectively to video summarization on small datasets?
• RQ5 Does the proposed method outperform existing ranking-based or segment-level approaches in terms of user-specific accuracy and computational efficiency?

Key findings

• Adaptive-H-FCSN achieves a 16.73% F-score on the PHD-GIFs dataset, significantly outperforming the generic H-FCSN (15.04%) and H-FCSN-aggregate (15.73%) models.
• Even with only one highlight in a user’s history, Adaptive-H-FCSN outperforms the generic H-FCSN, demonstrating the effectiveness of user history with minimal data.
• Performance of Adaptive-H-FCSN improves steadily with increasing history size, reaching 16.73% F-score when full history is used, indicating strong scalability with more user data.
• The model achieves state-of-the-art performance on the SumMe video summarization dataset (44.4% F-score) without any fine-tuning, outperforming supervised methods that require training on SumMe.
• The ablation study confirms that T-AIN outperforms direct feature fusion and other normalization strategies, validating its design for user-adaptive learning.