[Paper Review] Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling

Rui Wang, Zuxuan Wu+7 more|arXiv (Cornell University)|Aug 25, 2022

Advanced Neural Network Applications4 citations

TL;DR

Video Mobile-Former proposes a lightweight video recognition model that decouples local feature extraction via a 3D-CNN backbone from global spatial-temporal modeling using only six learnable global tokens in a Transformer encoder. By employing cross-attention between these global tokens and local features, it achieves state-of-the-art performance at under 1G FLOPs, outperforming both efficient CNNs and larger video Transformers in the low FLOP regime.

ABSTRACT

Transformer-based models have achieved top performance on major video recognition benchmarks. Benefiting from the self-attention mechanism, these models show stronger ability of modeling long-range dependencies compared to CNN-based models. However, significant computation overheads, resulted from the quadratic complexity of self-attention on top of a tremendous number of tokens, limit the use of existing video transformers in applications with limited resources like mobile devices. In this paper, we extend Mobile-Former to Video Mobile-Former, which decouples the video architecture into a lightweight 3D-CNNs for local context modeling and a Transformer modules for global interaction modeling in a parallel fashion. To avoid significant computational cost incurred by computing self-attention between the large number of local patches in videos, we propose to use very few global tokens (e.g., 6) for a whole video in Transformers to exchange information with 3D-CNNs with a cross-attention mechanism. Through efficient global spatial-temporal modeling, Video Mobile-Former significantly improves the video recognition performance of alternative lightweight baselines, and outperforms other efficient CNN-based models at the low FLOP regime from 500M to 6G total FLOPs on various video recognition tasks. It is worth noting that Video Mobile-Former is the first Transformer-based video model which constrains the computational budget within 1G FLOPs.

Motivation & Objective

• To design a video recognition model that maintains high accuracy while operating under strict computational constraints, such as those on mobile devices.
• To overcome the quadratic computational complexity of standard video Transformers by drastically reducing the number of tokens processed by self-attention mechanisms.
• To integrate the strengths of 3D-CNNs (local inductive bias) and Transformers (global long-range modeling) in a lightweight, efficient architecture.
• To enable real-time inference on mobile and CPU platforms by minimizing FLOPs and latency.

Proposed method

• The model uses a two-branch architecture: a Mobile branch based on lightweight 3D depthwise and pointwise convolutions for local context modeling.
• A Former branch employs multi-head self-attention on only six learnable global tokens to model global spatial-temporal dependencies across the entire video.
• Cross-attention mechanisms are used to enable bidirectional information exchange between the global tokens and local 3D-CNN features.
• Temporal downsampling is applied early in the network via a large temporal stride in the convolutional stem to reduce sequence length and computational cost.
• Dynamic ReLU activation is introduced in the Mobile block, using frame-level features to generate channel-wise parameters, improving representational capacity.
• The architecture is designed to maintain high accuracy at low FLOP regimes (500M to 6G), with a focus on inference efficiency on CPU and mobile hardware.

Experimental results

Research questions

• RQ1 Can a video Transformer be made efficient enough for mobile deployment while maintaining strong performance?
• RQ2 Does using only six global tokens in self-attention enable effective global modeling of long-range spatial-temporal dependencies in videos?
• RQ3 How does the combination of 3D-CNNs and a minimal Transformer module compare to state-of-the-art lightweight CNNs and full video Transformers in terms of accuracy and FLOPs?
• RQ4 What is the optimal placement and design of temporal downsampling in a lightweight video model?
• RQ5 Can dynamic activation functions improve performance in video recognition models with limited parameters?

Key findings

• Video Mobile-Former achieves 67.4% top-1 accuracy on Kinetics-400 with only 1G FLOPs, outperforming all other models in the same FLOP regime.
• The model runs in 29.0ms on a V100 GPU and 603.5ms on a CPU, significantly faster than comparable video Transformers and competitive with 3D-CNN baselines.
• Using only six global tokens achieves better performance than using more tokens or applying self-attention per frame, confirming the efficiency of the global token design.
• The ablation study shows that 3D depthwise convolutions in the Mobile block yield better performance than 2D or (2+1)D convolutions, despite a slight increase in FLOPs.
• Dynamic ReLU with frame-level feature modulation improves top-1 accuracy by 2.5 percentage points over ReLU and 1.8 points over the original Mobile-Former design.
• The model maintains strong performance even at shallow depths, with performance gains increasing as the network becomes shallower, indicating high efficiency for lightweight architectures.