[Paper Review] Space-time Mixing Attention for Video Transformer

Adrian Bulat, Juan-Manuel Pérez-Rúa+3 more|arXiv (Cornell University)|Jun 10, 2021

Advanced Vision and ImagingComputer Science45 references57 citations

TL;DR

Proposes a Video Transformer with space-time mixing attention that achieves linear scaling with the number of frames, by restricting temporal attention to a local window and mixing space-time information efficiently, plus lightweight temporal aggregation mechanisms.

ABSTRACT

This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer's depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models. Code will be made available.

Motivation & Objective

• Address the computational bottleneck of full space-time attention in video transformers.
• Propose a local-temporal window and space-time mixing mechanism to achieve O(TS^2) complexity.
• Maintain or improve accuracy while reducing FLOPs compared to contemporaneous Video Transformer approaches.
• Enable efficient temporal aggregation through lightweight global temporal attention and summary tokens.

Proposed method

• Compute patch-based token embeddings per frame with spatial and temporal positional encodings.
• Introduce local temporal window attention: y^l_{s,t} = sum_{t' in window} sum_s' Softmax(q^l_{s,t}·k^l_{s',t'}) v^l_{s',t'}.
• Apply the shift trick to implement space-time mixing by constructing a single attended key vector from multiple adjacent frames (Eq. 9–10).
• Achieve complexity O(TS^2) per layer, matching spatial-only attention costs.
• Optionally integrate lightweight global temporal attention (Temporal Attention) on class tokens to improve accuracy.
• Explore a Summary token mechanism to summarize frame information and attend to it during key/value computation.

Experimental results

Research questions

• RQ1 Can local temporal windows plus space-time mixing approximate full space-time attention efficiently in video transformers?
• RQ2 Does the proposed attention scheme maintain competitive accuracy while keeping or reducing FLOPs compared to existing Video Transformers?
• RQ3 What is the impact of local window size, ta layer placement, and temporal aggregation on performance?
• RQ4 Do lightweight temporal aggregation and summary tokens provide meaningful gains with minimal cost?

Key findings

• Achieves O(TS^2) complexity, matching spatial-only attention, while outperforming other efficiency-focused Video Transformers.
• Best local window size t_w = 1 among tested settings, with significant gains over t_w = 0 (spatial-only) and some degradation at t_w = 2 due to boundary effects.
• Space-time mixing applied to key and value vectors yields the strongest accuracy gains among mixing variants.
• Temporal aggregation using one Temporal Attention layer improves Top-1 by ~2 percentage points over simple temporal averaging; more TA layers do not help further.
• On SSv2, Kinetics-400, Kinetics-600, and Epic Kitchens, X-ViT matches or surpasses state-of-the-art while being substantially more efficient than TimeSformer and ViViT variants.
• Increasing input frames from 8 to 16 yields an ~1.8% Top-1 gain on SSv2; 32 frames provide a smaller additional gain.