[Paper Review] NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Amir Shahroudy, Jun Liu+2 more|arXiv (Cornell University)|Apr 11, 2016

Human Pose and Action RecognitionComputer Science47 references373 citations

TL;DR

Introduces NTU RGB+D, a large-scale RGB+D action recognition dataset with 56,880 samples across 60 classes and proposes a part-aware LSTM (P-LSTM) that outperforms baselines on cross-subject and cross-view evaluations.

ABSTRACT

Recent approaches in depth-based human activity analysis achieved outstanding performance and proved the effectiveness of 3D representation for classification of action classes. Currently available depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of training samples, distinct class labels, camera views and variety of subjects. In this paper we introduce a large-scale dataset for RGB+D human action recognition with more than 56 thousand video samples and 4 million frames, collected from 40 distinct subjects. Our dataset contains 60 different action classes including daily, mutual, and health-related actions. In addition, we propose a new recurrent neural network structure to model the long-term temporal correlation of the features for each body part, and utilize them for better action classification. Experimental results show the advantages of applying deep learning methods over state-of-the-art hand-crafted features on the suggested cross-subject and cross-view evaluation criteria for our dataset. The introduction of this large scale dataset will enable the community to apply, develop and adapt various data-hungry learning techniques for the task of depth-based and RGB+D-based human activity analysis.

Motivation & Objective

• Motivate the need for a large-scale, varied RGB+D action recognition benchmark to enable data-hungry learning methods.
• Provide a richly annotated dataset with diverse subjects, views, and environments to enable robust cross-subject and cross-view evaluations.
• Propose a part-aware LSTM model that leverages body-part structure to improve sequential action recognition.
• Assess traditional hand-crafted features, skeleton-based methods, and recurrent neural networks on the new dataset.
• Demonstrate that data-driven learning methods outperform hand-crafted features on the NTU RGB+D benchmark.

Proposed method

• Introduce NTU RGB+D data modalities: depth maps, 3D joints, RGB frames, and infrared sequences captured with Kinect v2.
• Collect 56,880 RGB+D video samples from 40 subjects, 80 camera viewpoints, and 60 action classes across daily, health-related, and mutual actions.
• Define cross-subject and cross-view evaluation criteria and provide preprocessed skeletal representations for learning.
• Develop Part-Aware LSTM (P-LSTM) by dividing the body into five parts and assigning part-specific memory cells while sharing the output gate.
• Formulate P-LSTM equations: per-part gates i^p, f^p, g^p with c_t^p memories; shared output gate o; concatenated part memories for h_t (Equations 9–12 in the paper).
• Compare P-LSTM against depth-map baselines, skeleton-based methods, and standard RNN/LSTM architectures.

Experimental results

Research questions

• RQ1 Can a large-scale RGB+D action dataset support data-hungry learning methods and improve cross-subject and cross-view generalization?
• RQ2 Do skeleton and depth-based features alone suffice, or can deep sequence models (RNN/LSTM) outperform hand-crafted features on NTU RGB+D?
• RQ3 Does incorporating body-part structure via a Part-Aware LSTM improve action recognition performance over standard LSTM on depth/skeleton data?
• RQ4 How do cross-subject and cross-view evaluations differ for various methods on the NTU RGB+D benchmark?
• RQ5 What is the relative performance gain of multi-layer P-LSTM versus single-layer and other recurrent architectures?

Key findings

• NTU RGB+D contains 56,880 samples, 60 action classes, 40 subjects, and 80 viewpoints, enabling broad cross-subject and cross-view evaluation.
• Part-Aware LSTM (P-LSTM) outperforms all baselines, achieving 62.93% cross-subject and 70.27% cross-view accuracy.
• Depth-map baselines perform worse in cross-view than cross-subject settings due to view-dependent depth appearance, while skeleton-based features generalize better across views but depend on accurate body tracking.
• Stacked LSTMs improve performance; P-LSTM provides the best results by modeling per-part dynamics and combining them.
• Compared to hand-crafted features (HOG2, HON4D, etc.) and other skeleton-based methods, P-LSTM shows clear advantages on the NTU RGB+D benchmark.