[Paper Review] VidTr: Video Transformer Without Convolutions

Yanyi Zhang, Xinyu Li+7 more|arXiv (Cornell University)|Apr 23, 2021

Human Pose and Action Recognition61 references4 citations

TL;DR

VidTr proposes a convolution-free video transformer with separable-attention for efficient and effective spatio-temporal modeling in video action recognition. By decoupling spatial and temporal attention and introducing standard deviation-based top-K pooling, VidTr reduces memory usage by 3.3× and FLOPs by up to 20× while achieving state-of-the-art performance on five datasets, especially excelling in long-term temporal reasoning tasks.

ABSTRACT

We introduce Video Transformer (VidTr) with separable-attention for video classification. Comparing with commonly used 3D networks, VidTr is able to aggregate spatio-temporal information via stacked attentions and provide better performance with higher efficiency. We first introduce the vanilla video transformer and show that transformer module is able to perform spatio-temporal modeling from raw pixels, but with heavy memory usage. We then present VidTr which reduces the memory cost by 3.3$ imes$ while keeping the same performance. To further optimize the model, we propose the standard deviation based topK pooling for attention ($pool_{topK\_std}$), which reduces the computation by dropping non-informative features along temporal dimension. VidTr achieves state-of-the-art performance on five commonly used datasets with lower computational requirement, showing both the efficiency and effectiveness of our design. Finally, error analysis and visualization show that VidTr is especially good at predicting actions that require long-term temporal reasoning.

Motivation & Objective

• To address the limitations of 3D convolutional networks in video action recognition, such as limited receptive fields and slow feature aggregation.
• To explore the feasibility of using pure transformers on raw video pixels for end-to-end spatio-temporal modeling without convolutional backbones.
• To reduce the high memory and computational cost of vanilla video transformers through architectural innovations like separable-attention and dynamic sequence pruning.
• To improve performance on long-range temporal reasoning tasks where global context is critical, especially for actions requiring extended temporal understanding.

Proposed method

• Proposes a vanilla video transformer that directly processes raw video pixels using vision transformer (ViT) principles, demonstrating the potential of self-attention for end-to-end video classification.
• Introduces separable-attention by decoupling spatial and temporal attention heads, reducing memory consumption by 3.3× with no accuracy drop compared to standard video transformers.
• Designs a standard deviation-based top-K pooling mechanism (pool_topK_std) that selects representative frames by filtering out low-variance, redundant temporal features.
• Employs a two-stage training strategy: pre-train on Kinetics-400, then fine-tune on downstream datasets, enabling strong generalization across diverse benchmarks.
• Uses attention rollout visualization to interpret attention patterns, showing that spatial attention focuses on action-relevant regions and temporal attention suppresses redundant frames.
• Enables model ensembling with I3D to combine global (transformer) and local (convolutional) modeling strengths, achieving +2% accuracy gain on Kinetics-400.

Experimental results

Research questions

• RQ1 Can a pure transformer-based architecture achieve state-of-the-art performance in video action recognition without any convolutional operations?
• RQ2 How can the high memory and FLOP cost of vanilla video transformers be reduced while preserving performance?
• RQ3 Does the proposed separable-attention mechanism effectively decouple spatial and temporal modeling, improving efficiency without sacrificing accuracy?
• RQ4 Can standard deviation-based top-K pooling improve efficiency by removing redundant temporal frames, and does it preserve or enhance model performance?
• RQ5 Does the VidTr model show superior performance on actions requiring long-term temporal reasoning compared to convolutional baselines like I3D?

Key findings

• VidTr-S achieves state-of-the-art performance on Kinetics-400 and Kinetics-700 with 5× fewer FLOPs than I3D101-NL and 12% fewer than Slowfast101-8×8.
• The model reduces memory usage by 3.3× through separable-attention while maintaining the same accuracy as the vanilla video transformer.
• On Charades, VidTr-L achieves comparable performance to Slowfast101-NL and outperforms LFB and NUTA101, demonstrating strong generalization to multi-label datasets.
• On UCF-101 and HMDB-51, VidTr achieves 96.6% top-1 accuracy on UCF-101 and 74.4% on HMDB-51 with only 6 epochs of training, showing strong generalization on small datasets.
• Error analysis confirms VidTr significantly outperforms I3D on long-horizon actions like 'making a cake' vs. 'eating a cake', where temporal context is critical.
• Ensembling VidTr-L with I3D50 leads to a 0.6% performance boost on Kinetics-700, confirming complementary strengths between global (transformer) and local (convolutional) modeling.