[Paper Review] Tripping through time: Efficient Localization of Activities in Videos

Meera Hahn, Asim Kadav+2 more|arXiv (Cornell University)|Apr 22, 2019

Multimodal Machine Learning Applications20 references41 citations

TL;DR

TripNet localizes moments in untrimmed videos by grounding natural language queries using a gated-attention representation and a reinforcement-learning-based search that probes only 32-41% of the video, achieving strong accuracy.

ABSTRACT

Localizing moments in untrimmed videos via language queries is a new and interesting task that requires the ability to accurately ground language into video. Previous works have approached this task by processing the entire video, often more than once, to localize relevant activities. In the real world applications of this approach, such as video surveillance, efficiency is a key system requirement. In this paper, we present TripNet, an end-to-end system that uses a gated attention architecture to model fine-grained textual and visual representations in order to align text and video content. Furthermore, TripNet uses reinforcement learning to efficiently localize relevant activity clips in long videos, by learning how to intelligently skip around the video. It extracts visual features for few frames to perform activity classification. In our evaluation over Charades-STA, ActivityNet Captions and the TACoS dataset, we find that TripNet achieves high accuracy and saves processing time by only looking at 32-41% of the entire video.

Motivation & Objective

• Address the challenge of temporally localizing actions described by natural language in long, untrimmed videos.
• Develop an end-to-end framework that grounds language to fine-grained video features.
• Improve efficiency by learning a policy that intelligently skips non-essential frames.

Proposed method

• Propose TripNet with a gated-attention state representation that aligns language queries with video features.
• Use an actor-critic reinforcement learning framework (A3C) to learn a policy for moving a fixed-size candidate window over the video.
• Define a discrete action space to jump the window by predefined frame steps and a TERMINATE action to output the current window.
• Incorporate a reward that combines improvement in intersection-over-union (IOU) with a small penalty for the number of steps to encourage efficiency.
• Train the model end-to-end so that visual and textual modalities are fused before policy learning.
• Compare gated-attention TripNet with a concatenation-baseline TripNet-Concat to demonstrate the benefit of gated attention.

Experimental results

Research questions

• RQ1 Can TripNet accurately localize moments described by natural language in long videos?
• RQ2 Does a gated-attention fusion model improve grounding accuracy over simple feature concatenation?
• RQ3 How much video can be skipped while still achieving strong localization performance?
• RQ4 How does TripNet compare with prior TALL methods in terms of accuracy and efficiency on standard benchmarks?

Key findings

• TripNet achieves state-of-the-art or competitive accuracy on Charades-STA, ActivityNet Captions, and TACoS datasets.
• TripNetLocalizes moments while inspecting only 32-41% of the video on average, significantly increasing efficiency.
• TripNet-GA (gated attention) outperforms TripNet-Concat, demonstrating the effectiveness of multi-modal gated fusion.
• On Charades-STA and TACoS, TripNet outperforms prior methods; on ActivityNet Captions, it is comparable to the state of the art.
• The approach reduces overall video processing time by avoiding exhaustive frame-by-frame analysis while maintaining high localization accuracy.