[Paper Review] TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding

Zhengwei Wang, Qi She+1 more|arXiv (Cornell University)|Oct 17, 2021

Human Pose and Action Recognition27 references4 citations

TL;DR

TEAM-Net proposes a plug-and-play multi-modal learning module (TEAM) for video action recognition using partially decoded videos, leveraging I-frames, motion vectors (MVs), and residuals from Group of Pictures (GOPs). By fusing these modalities end-to-end through sequential channel and spatial fusion, it achieves state-of-the-art performance on UCF-101 and HMDB-51, outperforming RGB-only baselines and prior compressed-domain methods.

ABSTRACT

Most of existing video action recognition models ingest raw RGB frames. However, the raw video stream requires enormous storage and contains significant temporal redundancy. Video compression (e.g., H.264, MPEG-4) reduces superfluous information by representing the raw video stream using the concept of Group of Pictures (GOP). Each GOP is composed of the first I-frame (aka RGB image) followed by a number of P-frames, represented by motion vectors and residuals, which can be regarded and used as pre-extracted features. In this work, we 1) introduce sampling the input for the network from partially decoded videos based on the GOP-level, and 2) propose a plug-and-play mulTi-modal lEArning Module (TEAM) for training the network using information from I-frames and P-frames in an end-to-end manner. We demonstrate the superior performance of TEAM-Net compared to the baseline using RGB only. TEAM-Net also achieves the state-of-the-art performance in the area of video action recognition with partial decoding. Code is provided at https://github.com/villawang/TEAM-Net.

Motivation & Objective

• To address the high storage and redundancy in raw video data by leveraging partially decoded video streams from compressed domains.
• To improve video action recognition performance by effectively fusing three modalities—appearance (I-frames), motion (MVs), and residual texture—within a unified framework.
• To overcome limitations of prior methods that train separate backbones and fuse only at inference, by enabling end-to-end training with mid-level fusion.
• To propose a plug-and-play module (TEAM) that enables efficient, effective multi-modal learning in the partially compressed domain.

Proposed method

• Sampling inputs at the GOP level, using the first I-frame and one P-frame (with MVs and residuals) per GOP instead of full RGB decoding.
• Introducing a multi-modal learning module (TEAM) that performs sequential fusion: first channel-wise fusion to capture global modality importance, then spatial fusion to localize relevant regions.
• Designing five configurations of TEAM, with the best-performing variant being channel fusion followed by spatial fusion (TEAM_{c→s}).
• Embedding the TEAM module at multiple stages (after Conv1, Res2, Res3, Res4) in the TSN backbone to enable mid-level fusion and avoid degradation from late-stage fusion.
• Using a dual-branch feature aggregation strategy where each modality is processed independently before fusion, preserving modality-specific representations.
• Training the entire network end-to-end using cross-entropy loss, with no separate training for individual modalities.

Experimental results

Research questions

• RQ1 Can multi-modal learning using I-frames, motion vectors, and residuals from partially decoded videos improve video action recognition performance compared to RGB-only baselines?
• RQ2 Does end-to-end training with mid-level fusion of multiple modalities outperform late-fusion or separate backbone training in the partially compressed domain?
• RQ3 What is the optimal fusion order between channel and spatial fusion for multi-modal feature integration in video action recognition?
• RQ4 How does the choice of embedding location (stage) within the network affect the performance of the TEAM module?
• RQ5 Can a plug-and-play module effectively integrate multiple modalities without architectural redesign?

Key findings

• TEAM-Net achieves 94.3% average accuracy on UCF-101 and 73.8% on HMDB-51 using 8-frame inputs, outperforming the RGB-only TSN baseline.
• The sequential fusion strategy (TEAM_{c→s}) yields the best performance, with a 0.6% absolute gain over the parallel configuration on UCF-101.
• Fusing at mid-level stages (after Conv1, Res2, Res3, Res4) yields superior performance compared to late-stage fusion, which degrades accuracy due to low spatial resolution.
• The TEAM module successfully captures both 'what' (via channel fusion) and 'where' (via spatial fusion), as confirmed by class-specific attention visualizations.
• The method achieves state-of-the-art performance in the partially compressed domain, surpassing prior works that use separate backbones or late fusion.
• The ablation study confirms that both channel and spatial fusion are necessary, as single-modality fusion (channel-only or spatial-only) underperforms the full TEAM module.