[Paper Review] TSM: Temporal Shift Module for Efficient Video Understanding

Ji Lin, Chuang Gan+1 more|arXiv (Cornell University)|Nov 20, 2018

Human Pose and Action RecognitionComputer Science68 references166 citations

TL;DR

TSM introduces a lightweight Temporal Shift Module that enables temporal modeling inside 2D CNNs by shifting feature channels across time, achieving 3D-CNN-like accuracy with 2D-CNN complexity, and extending to online low-latency video tasks.

ABSTRACT

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github.com/mit-han-lab/temporal-shift-module.

Motivation & Objective

• Motivate efficient video understanding that combines accuracy with low computation cost for real-world deployment.
• Develop a temporal modeling mechanism that incurs zero additional computation and parameters when integrated with 2D CNNs.
• Address both offline high-accuracy and online low-latency video recognition scenarios.

Proposed method

• Propose Temporal Shift Module (TSM) that shifts a subset of channels along the temporal dimension to mix information from neighboring frames.
• Incorporate TSM inside the residual branch (residual shift) to preserve current-frame spatial learning while enabling temporal fusion.
• Adopt bi-directional TSM for offline video understanding and uni-directional TSM for online real-time processing.
• Apply partial shift (e.g., 1/4 channels) to minimize data movement and latency while maintaining temporal modeling capacity.
• Demonstrate hardware efficiency by keeping computation and parameters equal to the 2D CNN backbone and showcasing edge-device applicability.

Experimental results

Research questions

• RQ1 How can temporal information be incorporated into 2D CNNs without adding computation or parameters?
• RQ2 What is the impact of shifting only a subset of channels and embedding the shift within residual blocks on accuracy and efficiency?
• RQ3 Can the proposed TSM enable both offline high-accuracy and online low-latency video understanding on edge devices?

Key findings

• TSM improves 2D CNN baselines by large margins on temporally-focused datasets with zero extra computation.
• Bi-directional TSM achieves state-of-the-art results on Something-Something datasets while maintaining 2D CNN efficiency.
• Uni-directional TSM enables online low-latency video recognition with minimal memory and almost no additional latency.
• TSM offers strong hardware efficiency with favorable accuracy-FLOPs trade-offs compared to 3D-CNN and other efficient video models.
• TSM generalizes to online video object detection, improving mAP over 2D baselines with negligible latency.
• Edge deployments show practical latency reductions on devices like Jetson Nano and Galaxy Note8.