[Paper Review] Cooperative Cross-Stream Network for Discriminative Action Representation

Jingran Zhang, Fumin Shen+2 more|arXiv (Cornell University)|Aug 27, 2019

Human Pose and Action Recognition47 references4 citations

TL;DR

This paper proposes a Cooperative Cross-Stream (CCS) network that jointly optimizes spatial (RGB) and temporal (optical flow) stream features via a modality complementarity block and a joint loss function combining cross-entropy and modality ranking constraints. The method enhances discriminative feature learning and reduces modality discrepancy, achieving state-of-the-art performance on HMDB-51, UCF-101, and Kinetics-S something datasets.

ABSTRACT

Spatial and temporal stream model has gained great success in video action recognition. Most existing works pay more attention to designing effective features fusion methods, which train the two-stream model in a separate way. However, it's hard to ensure discriminability and explore complementary information between different streams in existing works. In this work, we propose a novel cooperative cross-stream network that investigates the conjoint information in multiple different modalities. The jointly spatial and temporal stream networks feature extraction is accomplished by an end-to-end learning manner. It extracts this complementary information of different modality from a connection block, which aims at exploring correlations of different stream features. Furthermore, different from the conventional ConvNet that learns the deep separable features with only one cross-entropy loss, our proposed model enhances the discriminative power of the deeply learned features and reduces the undesired modality discrepancy by jointly optimizing a modality ranking constraint and a cross-entropy loss for both homogeneous and heterogeneous modalities. The modality ranking constraint constitutes intra-modality discriminative embedding and inter-modality triplet constraint, and it reduces both the intra-modality and cross-modality feature variations. Experiments on three benchmark datasets demonstrate that by cooperating appearance and motion feature extraction, our method can achieve state-of-the-art or competitive performance compared with existing results.

Motivation & Objective

• To address the limitation of separate training in two-stream networks, which fails to exploit complementary spatial-temporal information and suffers from modality discrepancy.
• To improve discriminative power of deep features by jointly optimizing modality-specific and cross-modality constraints.
• To reduce intra- and inter-modality feature variations through a novel ranking loss that enforces triplet and embedding constraints.
• To enable end-to-end training of spatial and temporal streams with mutual enhancement, improving classification of ambiguous actions like 'drink' vs. 'eat'.

Proposed method

• Introduces a connection block to explicitly model cross-modality correlations between RGB and optical flow features.
• Employs a joint loss function combining standard cross-entropy loss with a modality ranking constraint for both homogeneous and heterogeneous modality learning.
• The modality ranking constraint includes intra-modality discriminative embedding and inter-modality triplet loss to minimize feature variance within and across modalities.
• Uses element-wise average for late fusion of clip-level scores, which outperforms concatenation and other fusion strategies on HMDB-51.
• Employs backbone networks like BN-Inception, ResNet, and VGG pre-trained on ImageNet for feature extraction.
• Applies Class Activation Mapping (CAM) to visualize attention regions, showing the model focuses on action-relevant parts like hands and eyes.

Experimental results

Research questions

• RQ1 Can joint end-to-end training of RGB and optical flow streams improve action recognition accuracy compared to separate training?
• RQ2 How can modality discrepancy between appearance and motion features be reduced to enhance feature discriminability?
• RQ3 Does a joint loss combining cross-entropy and modality ranking improve performance on ambiguous action classes like 'drink' and 'eat'?
• RQ4 What is the optimal fusion strategy for clip-level scores from two-stream networks in terms of accuracy and robustness?

Key findings

• The proposed CCS network achieves state-of-the-art or competitive performance on HMDB-51, UCF-101, and Kinetics-S something datasets.
• Element-wise average fusion of clip scores outperforms concatenation and other late fusion methods, achieving the highest accuracy on HMDB-51.
• Setting hyperparameters α₁=0.3, α₂=0.3, and α₃=0.8 yields the best accuracy of 97.4% on UCF-101, demonstrating optimal trade-off between convergence and performance.
• BN-Inception backbone achieves the best performance among VGG, ResNet, and BN-Inception on UCF-101, highlighting its suitability for the proposed framework.
• CAM visualization confirms that the model learns to focus on discriminative body parts such as hands and eyes, aligning with human perception of action-relevant regions.
• The modality ranking constraint effectively reduces both intra-modality and inter-modality feature variations, improving generalization and robustness.