[Paper Review] What and When to Look?: Temporal Span Proposal Network for Video Visual Relation Detection.

Sangmin Woo, Junhyug Noh+1 more|arXiv (Cornell University)|Jul 15, 2021

Multimodal Machine Learning Applications22 references4 citations

TL;DR

This paper proposes the Temporal Span Proposal Network (TSPN), a novel method for Video Visual Relation Detection that simultaneously identifies *what* relations exist between object pairs and *when* they occur by sparsifying the relation search space and leveraging full video context for temporal span and category prediction. TSPN achieves state-of-the-art performance on ImageNet-VidVDR and VidOR while being twice as efficient as a leading segment-based method.

ABSTRACT

Identifying relations between objects is central to understanding the scene. While several works have been proposed for relation modeling in the image domain, there have been many constraints in the video domain due to challenging dynamics of spatio-temporal interactions (e.g., Between which objects are there an interaction? When do relations occur and end?). To date, two representative methods have been proposed to tackle Video Visual Relation Detection (VidVRD): segment-based and window-based. We first point out the limitations these two methods have and propose Temporal Span Proposal Network (TSPN), a novel method with two advantages in terms of efficiency and effectiveness. 1) TSPN tells what to look: it sparsifies relation search space by scoring relationness (i.e., confidence score for the existence of a relation between pair of objects) of object pair. 2) TSPN tells when to look: it leverages the full video context to simultaneously predict the temporal span and categories of the entire relations. TSPN demonstrates its effectiveness by achieving new state-of-the-art by a significant margin on two VidVRD benchmarks (ImageNet-VidVDR and VidOR) while also showing lower time complexity than existing methods - in particular, twice as efficient as a popular segment-based approach.

Motivation & Objective

• To address the challenge of identifying both *which* object pairs interact and *when* these interactions occur in videos, due to complex spatio-temporal dynamics.
• To overcome limitations in existing segment-based and window-based methods for Video Visual Relation Detection (VidVRD), which suffer from inefficiency and incomplete modeling of temporal context.
• To propose a unified framework that jointly predicts relation categories and their temporal spans while reducing the search space through relationness scoring.
• To achieve higher accuracy and lower time complexity than prior state-of-the-art methods in VidVRD benchmarks.

Proposed method

• TSPN introduces a relationness scoring mechanism that evaluates the confidence of a relation existing between a pair of objects, thereby sparsifying the relation search space.
• It leverages the full video context to predict both the temporal span (start and end timestamps) and category of each detected relation in a single forward pass.
• The model uses a two-stream architecture to encode spatial and temporal features, enabling joint modeling of object interactions across time.
• Temporal span prediction is performed via regression heads that predict start and end times relative to the video's temporal extent.
• The framework integrates object detection and relation prediction in an end-to-end differentiable manner, allowing joint optimization.
• Relationness scores are used to filter low-confidence object pairs, reducing computational cost without sacrificing detection accuracy.

Experimental results

Research questions

• RQ1 How can we effectively reduce the search space for relations in videos while preserving detection accuracy?
• RQ2 What is the impact of leveraging full video context on temporal span and relation category prediction in VidVRD?
• RQ3 Can a unified framework jointly predict relation content and temporal duration more efficiently than existing segment-based or window-based approaches?
• RQ4 To what extent does the proposed relationness scoring improve the efficiency and effectiveness of VidVRD compared to prior methods?

Key findings

• TSPN achieves new state-of-the-art performance on both ImageNet-VidVDR and VidOR benchmarks, significantly outperforming prior methods.
• The method demonstrates a 2x speedup compared to a popular segment-based VidVRD approach, indicating superior efficiency.
• By leveraging full video context, TSPN improves temporal localization accuracy of relations, enabling precise prediction of when interactions occur.
• The relationness scoring mechanism effectively reduces the number of candidate relations, leading to lower time complexity without compromising detection quality.