[Paper Review] IntentVizor: Towards Generic Query Guided Interactive Video Summarization

Guande Wu, Jianzhe Lin+1 more|arXiv (Cornell University)|Sep 30, 2021

Video Analysis and Summarization4 citations

TL;DR

IntentVizor proposes an interactive, query-guided video summarization framework that uses multi-modal queries (text and video snippets) to represent user intent as a learnable, editable distribution over basis intents. By integrating a novel Granularity-Scalable Ego-Graph Convolutional Network (GSE-GCN) and a visual interface for real-time intent adjustment, the framework improves summary quality and interpretability, outperforming state-of-the-art methods on benchmark datasets with strong generalization across modalities.

ABSTRACT

The target of automatic video summarization is to create a short skim of the original long video while preserving the major content/events. There is a growing interest in the integration of user queries into video summarization or query-driven video summarization. This video summarization method predicts a concise synopsis of the original video based on the user query, which is commonly represented by the input text. However, two inherent problems exist in this query-driven way. First, the text query might not be enough to describe the exact and diverse needs of the user. Second, the user cannot edit once the summaries are produced, while we assume the needs of the user should be subtle and need to be adjusted interactively. To solve these two problems, we propose IntentVizor, an interactive video summarization framework guided by generic multi-modality queries. The input query that describes the user's needs are not limited to text but also the video snippets. We further represent these multi-modality finer-grained queries as user `intent', which is interpretable, interactable, editable, and can better quantify the user's needs. In this paper, we use a set of the proposed intents to represent the user query and design a new interactive visual analytic interface. Users can interactively control and adjust these mixed-initiative intents to obtain a more satisfying summary through the interface. Also, to improve the summarization quality via video understanding, a novel Granularity-Scalable Ego-Graph Convolutional Networks (GSE-GCN) is proposed. We conduct our experiments on two benchmark datasets. Comparisons with the state-of-the-art methods verify the effectiveness of the proposed framework. Code and dataset are available at https://github.com/jnzs1836/intent-vizor.

Motivation & Objective

• To address the limitations of static, text-only queries in query-driven video summarization, which often fail to capture nuanced or evolving user needs.
• To enable interactive, user-guided summarization where users can iteratively refine their intent through a visual interface.
• To develop a generic, multi-modal query framework that supports text, video snippets, and potentially other modalities for flexible user input.
• To improve video summarization performance by modeling user intent as a learnable, adjustable distribution over basis intents, enhancing interpretability and precision.
• To design a unified deep learning backbone (GSE-GCN) that supports both intent extraction and summary generation across different query modalities.

Proposed method

• The framework introduces 'intent' as a learnable, interpretable, and editable distribution over basis intents—low-level semantic components of user needs—instead of fixed categories or embeddings.
• A novel Granularity-Scalable Ego-Graph Convolutional Network (GSE-GCN) is proposed to jointly encode video features and multi-modal queries, enabling flexible temporal granularity and shared representation learning.
• The intent module transforms multi-modal queries (text or video snippets) into a probability distribution over basis intents using the GSE-GCN backbone.
• The summary module generates a concise video synopsis by combining the basis intents weighted by their probability distribution, using the same GSE-GCN backbone.
• A mixed-initiative visual interface allows users to interactively adjust the intent distribution in real time, with immediate feedback on the generated summary.
• The framework supports zero-shot transfer learning: a summary module pre-trained on text queries can be fine-tuned on visual queries without retraining the entire model.

Experimental results

Research questions

• RQ1 Can multi-modal queries (text and video snippets) better represent diverse and evolving user needs than text-only queries in video summarization?
• RQ2 How can user intent be modeled as an adjustable, interpretable distribution over basis intents to support interactive refinement?
• RQ3 To what extent can a shared GSE-GCN backbone effectively support both intent extraction and video summarization across different query modalities?
• RQ4 Does the interactive interface with real-time intent manipulation improve user satisfaction and summary quality compared to non-interactive baselines?
• RQ5 Can a model pre-trained on text queries generalize effectively to visual queries through transfer learning?

Key findings

• The proposed IntentVizor framework achieves state-of-the-art performance on two benchmark datasets, demonstrating superior effectiveness in query-guided video summarization.
• The transfer learning experiment shows that a summary module pre-trained on text queries generalizes well to visual queries, outperforming models trained in the canonical setting.
• The visual interface enables users to iteratively refine summaries by adjusting intent distributions, significantly improving user control and satisfaction.
• The ablation study confirms the necessity of the granularity-scalable design in GSE-GCN, showing improved performance across varying temporal granularities.
• Qualitative analysis confirms that the framework identifies fine-grained intent clusters, such as 'food cooking' and 'food storage', demonstrating interpretability and semantic richness.
• The framework successfully captures complex, multi-faceted user needs—e.g., distinguishing between 'dining table' and 'working table'—through disentangled basis intents.