[Paper Review] Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Mandela Patrick, Dylan Campbell+6 more|arXiv (Cornell University)|Jun 9, 2021

Human Pose and Action RecognitionComputer Science89 references49 citations

TL;DR

The paper introduces trajectory attention for video transformers, enabling pooling along motion trajectories, and proposes Orthoformer to efficiently approximate attention. It achieves state-of-the-art results on Kinetics, Something–Something V2, Epic-Kitchens, and related benchmarks.

ABSTRACT

In video transformers, the time dimension is often treated in the same way as the two spatial dimensions. However, in a scene where objects or the camera may move, a physical point imaged at one location in frame $t$ may be entirely unrelated to what is found at that location in frame $t+k$. These temporal correspondences should be modeled to facilitate learning about dynamic scenes. To this end, we propose a new drop-in block for video transformers -- trajectory attention -- that aggregates information along implicitly determined motion paths. We additionally propose a new method to address the quadratic dependence of computation and memory on the input size, which is particularly important for high resolution or long videos. While these ideas are useful in a range of settings, we apply them to the specific task of video action recognition with a transformer model and obtain state-of-the-art results on the Kinetics, Something--Something V2, and Epic-Kitchens datasets. Code and models are available at: https://github.com/facebookresearch/Motionformer

Motivation & Objective

• Motivate better temporal modeling in video transformers by capturing motion trajectories rather than treating time as a separate axis.
• Propose trajectory attention that aggregates information along implicitly determined motion paths in videos.
• Address the quadratic computation and memory cost of attention with an efficient approximation (Orthoformer).
• Demonstrate state-of-the-art performance on standard video action recognition benchmarks.
• Provide analysis and ablations to understand the impact of tokenization, positional encodings, and architectural choices.

Proposed method

• Introduce trajectory attention as a two-stage attention: (i) form trajectory tokens by spatial attention across frames, (ii) pool along these trajectories with a 1D temporal attention.
• Compute trajectory tokens via per-frame spatial attention: tilde{y}_{stt'} = sum_{s'} v_{s't'} * softmax(q_{st}^T k_{s't'}) / sum_{s''} softmax(q_{st}^T k_{s''t'}) .
• Project trajectory tokens to new queries/keys/values and apply 1D temporal attention across time: y_{st} = sum_{t'} tilde{v}_{stt'} * softmax(tilde{q}_{st}^T tilde{k}_{stt'}) / sum_{t''} softmax(tilde{q}_{st}^T tilde{k}_{st t''}).
• Propose Orthoformer, a low-cost attention approximation based on selecting a most-orthogonal subset of prototypes to reduce complexity from quadratic to near-linear in practice when prototypes are fixed.
• Share prototypes across time to exploit temporal redundancy in video data, enabling memory savings.
• Provide algorithms (Algorithm 1 and Algorithm 2) and compare with Nyströmformer and Performer, showing competitive or superior accuracy with fewer prototypes.

Experimental results

Research questions

• RQ1 How can attention in video transformers be biased to explicitly model motion trajectories rather than treating space and time equivalently?
• RQ2 Does trajectory-based pooling improve action recognition, especially on datasets with strong motion cues (e.g., Something–Something V2) compared to joint or divided space-time attention?
• RQ3 Can a prototype-based attention approximation (Orthoformer) provide substantial computational and memory efficiency while preserving accuracy for video data and beyond?
• RQ4 What is the impact of input tokenization (cubic vs. square) and spatial-temporal positional encoding on trajectory attention performance?
• RQ5 How does trajectory attention interact with common design choices (tokenization, positional encodings) across standard video transformer baselines?

Key findings

• Trajectory attention yields higher accuracy than joint or divided space-time attention on motion-sensitive datasets, with notable gains on Something–Something V2.
• The model demonstrates state-of-the-art results on multiple benchmarks, including Kinetics, Something–Something V2, and Epic-Kitchens.
• Orthoformer provides competitive or superior accuracy with significantly reduced memory and compute compared to exact attention and Nyströmformer across Long Range Arena tasks and video recognition benchmarks.
• Cubic tokenization and separated space-time positional encodings improve performance for trajectory attention, particularly on motion-rich datasets.
• Temporal sharing of prototypes and per-frame normalization contribute to efficiency and accuracy, with ablations showing losses when these design choices are altered.