[Paper Review] View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data

Pengfei Zhang, Cuiling Lan+4 more|arXiv (Cornell University)|Mar 24, 2017

Human Pose and Action RecognitionComputer Science39 references56 citations

TL;DR

Introduces a view-adaptive LSTM framework that automatically regresses observation viewpoints for skeleton-based action recognition, achieving state-of-the-art results on NTU, SBU, and SYSU datasets.

ABSTRACT

Skeleton-based human action recognition has recently attracted increasing attention due to the popularity of 3D skeleton data. One main challenge lies in the large view variations in captured human actions. We propose a novel view adaptation scheme to automatically regulate observation viewpoints during the occurrence of an action. Rather than re-positioning the skeletons based on a human defined prior criterion, we design a view adaptive recurrent neural network (RNN) with LSTM architecture, which enables the network itself to adapt to the most suitable observation viewpoints from end to end. Extensive experiment analyses show that the proposed view adaptive RNN model strives to (1) transform the skeletons of various views to much more consistent viewpoints and (2) maintain the continuity of the action rather than transforming every frame to the same position with the same body orientation. Our model achieves significant improvement over the state-of-the-art approaches on three benchmark datasets.

Motivation & Objective

• Motivate robust action recognition from 3D skeleton data under large viewpoint variations.
• Develop a view adaptation mechanism that regresses the observation viewpoint frame-by-frame.
• Integrate the view adaptation module with a main LSTM network in an end-to-end trainable system.
• Demonstrate improved recognition performance over state-of-the-art methods across multiple datasets.

Proposed method

• Propose a View Adaptation Subnetwork that predicts per-frame rotation (alpha, beta, gamma) and translation (d) to form a new observation coordinate system.
• Represent each frame's joints under the adapted viewpoint using a rigid-body transformation (R_t and d_t).
• Use separate LSTM branches to learn rotation and translation parameters from the frame input.
• Feed the adapted skeletons into a Main LSTM Network (stacked 3 LSTM layers) for temporal feature learning and classification.
• Train the entire network end-to-end with cross-entropy loss, propagating gradients from the Main LSTM back to the View Adaptation Subnetwork.

Experimental results

Research questions

• RQ1 Can a learnable, frame-wise viewpoint adaptation improve robustness to view changes in skeleton-based action recognition?
• RQ2 What is the impact of jointly learning viewpoint regulation and action classification on recognition accuracy?
• RQ3 How does VA-LSTM compare with traditional pre-processing-based view normalization approaches across benchmark datasets?
• RQ4 Does the view adaptation preserve motion continuity while aligning viewpoints across subjects and actions?

Key findings

• VA-LSTM achieves state-of-the-art accuracy on NTU with CS 79.4 and CV 87.6.
• On SBU, VA-LSTM reaches 97.2% accuracy, outperforming prior methods.
• On SYSU, VA-LSTM achieves 76.9% (setting-1) and 77.5% (setting-2).
• The end-to-end view adaptation consistently outperforms pre-processing strategies that rely on frame- or sequence-level rotations/translations.
• Visualization shows the model regresses skeletons to more consistent viewpoints while preserving action dynamics.