[Paper Review] Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos

Amir Shahroudy, Tian-Tsong Ng+2 more|arXiv (Cornell University)|Mar 23, 2016

Human Pose and Action RecognitionComputer Science78 references17 citations

TL;DR

This paper proposes a deep multimodal feature analysis framework, DSSCA, which hierarchically factorizes RGB+D video features into shared (common) and modality-specific components using a stacked autoencoder-based network. It further employs a structured sparsity learning machine (SSLM) with mixed-norm regularization to enhance classification by selecting informative components, achieving state-of-the-art accuracy across five RGB+D action recognition benchmarks, including 99.0% on the Online RGBD dataset.

ABSTRACT

Single modality action recognition on RGB or depth sequences has been extensively explored recently. It is generally accepted that each of these two modalities has different strengths and limitations for the task of action recognition. Therefore, analysis of the RGB+D videos can help us to better study the complementary properties of these two types of modalities and achieve higher levels of performance. In this paper, we propose a new deep autoencoder based shared-specific feature factorization network to separate input multimodal signals into a hierarchy of components. Further, based on the structure of the features, a structured sparsity learning machine is proposed which utilizes mixed norms to apply regularization within components and group selection between them for better classification performance. Our experimental results show the effectiveness of our cross-modality feature analysis framework by achieving state-of-the-art accuracy for action classification on five challenging benchmark datasets.

Motivation & Objective

• To address the limitations of single-modality action recognition by leveraging complementary information from RGB and depth modalities in RGB+D videos.
• To develop a deep neural network that hierarchically decomposes multimodal features into shared (common) and modality-specific components for improved robustness and discriminability.
• To design a structured sparsity-based learning machine that enables effective component and layer selection for enhanced classification performance.
• To demonstrate the superiority of the proposed framework over existing methods on multiple challenging RGB+D action recognition datasets.

Proposed method

• A deep autoencoder-based shared-specific feature factorization network is proposed, where each layer decomposes input RGB and depth features into shared components and modality-specific components.
• The network is stacked in a deep architecture to learn hierarchical, nonlinear representations of multimodal signals through end-to-end training.
• A structured sparsity learning machine (SSLM) is introduced, using mixed norms (e.g., ℓ2,1) to apply regularization within components and group selection across components and layers.
• The SSLM classifier learns to assign higher weights to more discriminative components, such as shared features and modality-specific features in deeper layers.
• The framework is trained end-to-end using backpropagation, with joint optimization of the factorization and classification objectives.
• Local and holistic features (e.g., HOG, HOF, HON4D) are extracted from RGB and depth streams before input to the network.

Experimental results

Research questions

• RQ1 Can a deep neural network effectively factorize multimodal RGB+D features into shared and modality-specific components while preserving hierarchical, nonlinear representations?
• RQ2 Does incorporating both shared and modality-specific components lead to better action recognition performance than using only shared components or single-modality features?
• RQ3 Can structured sparsity learning with mixed-norm regularization improve classification by enabling selective fusion of informative components across layers?
• RQ4 How does the performance of the proposed framework compare to state-of-the-art methods on standard RGB+D action recognition benchmarks?

Key findings

• The proposed DSSCA framework achieved state-of-the-art accuracy of 99.0% on the Online RGBD dataset, significantly outperforming previous methods.
• On the MSR Daily Activity 3D dataset, the method achieved 96.3% accuracy, surpassing single-modality baselines (89.4% for RGB, 92.5% for depth) and other multimodal approaches.
• The structured sparsity learning machine (SSLM) assigned high weights to shared components (e.g., Y³ with ℓ2 norm 0.20–0.42) and modality-specific components (Zr³, Zd³), confirming their discriminative value.
• The contribution analysis showed that some components received near-zero weights (e.g., Zd¹ in Online S3 with 0.00), indicating they were less relevant to action classification despite being present in the data.
• The performance gain of the framework was more pronounced in deeper networks, demonstrating the benefit of hierarchical feature decomposition.
• The method outperformed atomic local-level analysis by over 50% in error rate reduction on the Online RGBD dataset, confirming the advantage of stacked, deep factorization.