[Paper Review] More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation

Quanfu Fan, Chun-Fu Chen+3 more|arXiv (Cornell University)|Dec 2, 2019

Human Pose and Action RecognitionComputer Science90 citations

TL;DR

Introduces a lightweight, memory-efficient video architecture (bLVNet) with a dual-path Big-Little design and a compact temporal aggregation module (TAM) to model temporal relations without heavy 3D convolutions, achieving state-of-the-art results on Something-Something and Moments-in-Time while reducing FLOPs and memory.

ABSTRACT

Current state-of-the-art models for video action recognition are mostly based on expensive 3D ConvNets. This results in a need for large GPU clusters to train and evaluate such architectures. To address this problem, we present a lightweight and memory-friendly architecture for action recognition that performs on par with or better than current architectures by using only a fraction of resources. The proposed architecture is based on a combination of a deep subnet operating on low-resolution frames with a compact subnet operating on high-resolution frames, allowing for high efficiency and accuracy at the same time. We demonstrate that our approach achieves a reduction by $3\sim4$ times in FLOPs and $\sim2$ times in memory usage compared to the baseline. This enables training deeper models with more input frames under the same computational budget. To further obviate the need for large-scale 3D convolutions, a temporal aggregation module is proposed to model temporal dependencies in a video at very small additional computational costs. Our models achieve strong performance on several action recognition benchmarks including Kinetics, Something-Something and Moments-in-time. The code and models are available at https://github.com/IBM/bLVNet-TAM.

Motivation & Objective

• Reduce computational cost and memory footprint for video action recognition without sacrificing accuracy.
• Enable training with deeper backbones and more input frames under the same hardware budget.
• Develop a temporal aggregation mechanism that captures short- and long-range temporal dependencies efficiently.
• Facilitate effective temporal modeling without resorting to expensive 3D convolutions.

Proposed method

• Proposes Big-Little Video Net (bLVNet): a dual-path network where a deep, high-capacity branch processes low-resolution frames (Big-Net) and a compact branch processes high-resolution frames (Little-Net).
• Fuse the two branches at each layer to merge multi-scale features and enable efficient processing of more frames than a baseline TSN variant.
• Introduces a Temporal Aggregation Module (TAM): a lightweight, learnable, depthwise 1x1 convolution based module that performs channel-wise weighted aggregation across a temporal window to model short- and long-range dependencies.
• TAM operations include: (i) 1x1 depthwise convolutions to learn channel weights, (ii) temporal shifts of feature maps, (iii) aggregation across the temporal window with ReLU activation.
• The TAM is designed to be independent of spatial convolutions and adds negligible parameters and computation, and can be integrated with 2D or 3D backbones.

Experimental results

Research questions

• RQ1 Can a two-branch Big-Little network achieve comparable or better action recognition accuracy with reduced FLOPs and memory compared to 3D CNN baselines?
• RQ2 Does a lightweight temporal aggregation module (TAM) improve temporal modeling beyond local fusion in a dual-path video network?
• RQ3 How does increasing the number of input frames affect performance and efficiency in the proposed bLVNet-TAM architecture?
• RQ4 Is the TAM more effective for temporal modeling on challenging datasets (e.g., Something-Something) than existing temporal-shift methods?

Key findings

• bLVNet-TAM achieves strong performance with significantly lower FLOPs and memory than strong baselines, enabling deeper backbones and more input frames on a single compute node.
• Temporal Aggregation Module (TAM) provides clear gains over Temporal Shift Module (TSM) and complements local fusion, boosting Something-Something accuracy.
• On Something-Something, bLVNet-TAM with a deeper backbone (bLResNet-101) and many frames achieves new state-of-the-art results in RGB only setups.
• On Moments-in-Time, the approach outperforms single-stream and ensemble baselines in top-1 accuracy.
• Across benchmarks, more input frames generally improve performance for bLVNet-TAM, while memory usage remains favorable compared to TSN-based architectures.