[Paper Review] AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition

Yue Meng, Rameswar Panda+6 more|arXiv (Cornell University)|Feb 10, 2021

Human Pose and Action Recognition61 references21 citations

TL;DR

AdaFuse proposes an adaptive temporal fusion network that dynamically reuses historical feature maps and skips redundant channels during inference to achieve efficient video action recognition. By learning a differentiable policy via Gumbel-Softmax, it reduces FLOPs by ~40% with comparable accuracy to state-of-the-art models across multiple benchmarks.

ABSTRACT

Temporal modelling is the key for efficient video action recognition. While understanding temporal information can improve recognition accuracy for dynamic actions, removing temporal redundancy and reusing past features can significantly save computation leading to efficient action recognition. In this paper, we introduce an adaptive temporal fusion network, called AdaFuse, that dynamically fuses channels from current and past feature maps for strong temporal modelling. Specifically, the necessary information from the historical convolution feature maps is fused with current pruned feature maps with the goal of improving both recognition accuracy and efficiency. In addition, we use a skipping operation to further reduce the computation cost of action recognition. Extensive experiments on Something V1 & V2, Jester and Mini-Kinetics show that our approach can achieve about 40% computation savings with comparable accuracy to state-of-the-art methods. The project page can be found at https://mengyuest.github.io/AdaFuse/

Motivation & Objective

• Address the high computational cost of deep CNNs in video action recognition by exploiting feature redundancy across temporal frames.
• Develop a model-agnostic, differentiable mechanism to decide per-instance and per-layer whether to skip, reuse, or compute channels.
• Improve efficiency without sacrificing recognition accuracy by leveraging temporal coherence in video features.
• Provide a plug-in solution compatible with existing 2D-CNN architectures for action recognition.
• Enable insight into dataset-specific and layer-specific temporal dynamics through learned policy distributions.

Proposed method

• Introduce a Gumbel-Softmax-based differentiable decision policy to select among three operations: skip, reuse, or compute for each channel at each layer.
• Fuse current pruned features with historical feature maps using learned attention weights to preserve temporal context.
• Apply a skip connection to reduce computation by avoiding redundant feature computation in low-dynamic frames.
• Train the policy network jointly with the main network via back-propagation, using a weighted loss balancing accuracy and FLOP reduction.
• Design the policy network to be lightweight and scalable, with hidden size and regularization hyperparameters tuned for efficiency.
• Integrate AdaFuse as a plugin into existing 2D-CNN backbones like TSN, TSM, and ResNet, enabling plug-and-play efficiency gains.

Experimental results

Research questions

• RQ1 Can adaptive channel-level decisions (skip, reuse, compute) significantly reduce FLOPs in video action recognition without accuracy degradation?
• RQ2 How does the learned policy vary across different video datasets and network layers, and what does this reveal about temporal dynamics?
• RQ3 What is the trade-off between model size, inference speed, and accuracy when scaling the policy network's capacity?
• RQ4 How does the combination of skip and reuse operations compare to naive or random channel selection in terms of efficiency-accuracy trade-offs?
• RQ5 Can the policy distribution serve as a diagnostic tool for understanding dataset-specific temporal characteristics and guiding future architecture design?

Key findings

• AdaFuse achieves ~40% FLOP reduction on Something-Something V1 and V2, Jester, and Mini-Kinetics while maintaining state-of-the-art accuracy.
• The 'reuse' operation contributes most to accuracy gain, increasing Top-1 accuracy by 21.5 percentage points compared to naive skipping.
• The 'skip' operation is the primary driver of efficiency, reducing FLOPs by 55% in the 'Ada. Skip' ablation.
• Increasing the policy network's hidden size from 1024 to 4096 improves Top-1 accuracy from 59.40% to 60.00% on Something-V2 with minimal FLOP increase.
• The policy distribution reveals that earlier layers are less sensitive to temporal changes, while later layers adapt more dynamically to input dynamics.
• The adaptive policy outperforms random or fixed policies, achieving the best accuracy-efficiency trade-off with 10.3G FLOPs and 36.9% Top-1 accuracy on Something-V1 using TSN-R18.