[Paper Review] Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks

Wentao Zhu, Cuiling Lan+5 more|arXiv (Cornell University)|Mar 24, 2016

Human Pose and Action RecognitionComputer Science27 references289 citations

TL;DR

This paper proposes an end-to-end deep LSTM model for skeleton-based action recognition that learns joint co-occurrences with a specialized regularization, and applies in-depth dropout to LSTM neurons, achieving state-of-the-art results on multiple datasets.

ABSTRACT

Skeleton based action recognition distinguishes human actions using the trajectories of skeleton joints, which provide a very good representation for describing actions. Considering that recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) can learn feature representations and model long-term temporal dependencies automatically, we propose an end-to-end fully connected deep LSTM network for skeleton based action recognition. Inspired by the observation that the co-occurrences of the joints intrinsically characterize human actions, we take the skeleton as the input at each time slot and introduce a novel regularization scheme to learn the co-occurrence features of skeleton joints. To train the deep LSTM network effectively, we propose a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons. Experimental results on three human action recognition datasets consistently demonstrate the effectiveness of the proposed model.

Motivation & Objective

• Learn robust, discriminative features from skeleton joint trajectories for action recognition.
• Automatically discover and leverage co-occurring joint patterns across actions.
• Mitigate overfitting with novel regularization and an in-depth dropout strategy for LSTM neurons.
• Demonstrate superior performance on multiple skeleton-based action datasets.

Proposed method

• Use a fully connected deep LSTM network with three bidirectional LSTM layers and two feedforward layers.
• Introduce co-occurrence regularization to learn joint feature interactions, implemented via a group-wise (L21) norm and L1 sparsity terms in the loss.
• Apply co-occurrence regularization to connections feeding into the second LSTM layer to capture discriminative joint conjunctions.
• Develop an in-depth dropout mechanism for LSTM units that drops input gates, forget gates, cells, and output responses without erasing temporal memory.
• Train with a loss that combines maximum likelihood with regularization terms, and aggregate bidirectional LSTM outputs for final class probabilities.

Experimental results

Research questions

• RQ1 Does co-occurrence regularization improve discriminative feature learning for skeleton-based actions?
• RQ2 Does in-depth dropout on LSTM neurons enhance learning of complex temporal dynamics?
• RQ3 How does the proposed Deep LSTM with regularizations compare to state-of-the-art methods on standard skeleton-based action datasets?

Key findings

• Deep LSTM outperforms baseline approaches on three public datasets.
• Adding co-occurrence regularization improves accuracy (e.g., SBU Kinect: 86.03% to 89.44%).
• In-depth dropout yields further improvements over simple dropout (e.g., SBU Kinect: 89.70% to 90.10%).
• The combined use of co-occurrence regularization and in-depth dropout achieves the best performance (e.g., SBU Kinect: 90.41%).
• On HDM05, CMU datasets, and Berkeley MHAD, the proposed model consistently attains state-of-the-art or superior results, with the authors noting 100% accuracy on Berkeley MHAD.