[Paper Review] Learning Human Pose Models from Synthesized Data for Robust RGB-D Action Recognition

Jian Liu, Naveed Akhtar+1 more|arXiv (Cornell University)|Jul 4, 2017

Human Pose and Action Recognition62 references7 citations

TL;DR

This paper proposes Human Pose Models (HPMs) that learn viewpoint- and appearance-invariant representations of human poses from synthetically generated RGB-D data, using 3D human models, GANs to align synthetic with real image distributions, and CNNs to extract invariant features. The method achieves state-of-the-art performance on cross-view and cross-subject RGB-D action recognition benchmarks by leveraging large-scale, diverse synthetic training data and Fourier Temporal Pyramids for temporal modeling.

ABSTRACT

We propose Human Pose Models that represent RGB and depth images of human poses independent of clothing textures, backgrounds, lighting conditions, body shapes and camera viewpoints. Learning such universal models requires training images where all factors are varied for every human pose. Capturing such data is prohibitively expensive. Therefore, we develop a framework for synthesizing the training data. First, we learn representative human poses from a large corpus of real motion captured human skeleton data. Next, we fit synthetic 3D humans with different body shapes to each pose and render each from 180 camera viewpoints while randomly varying the clothing textures, background and lighting. Generative Adversarial Networks are employed to minimize the gap between synthetic and real image distributions. CNN models are then learned that transfer human poses to a shared high-level invariant space. The learned CNN models are then used as invariant feature extractors from real RGB and depth frames of human action videos and the temporal variations are modelled by Fourier Temporal Pyramid. Finally, linear SVM is used for classification. Experiments on three benchmark cross-view human action datasets show that our algorithm outperforms existing methods by significant margins for RGB only and RGB-D action recognition.

Motivation & Objective

• To address robust human action recognition under challenging variations such as viewpoint, lighting, clothing, and background.
• To develop a scalable method for generating large-scale, diverse synthetic RGB-D training data that captures action-irrelevant factors.
• To learn human pose representations invariant to viewpoint, body shape, texture, and illumination using deep CNNs.
• To model temporal dynamics of invariant pose features using a Fourier Temporal Pyramid for improved action classification.
• To demonstrate the effectiveness of synthetic data and invariant feature learning on benchmark RGB-D action recognition datasets.

Proposed method

• Representative human poses are extracted by clustering motion-captured skeleton data from the CMU MoCap database.
• Synthetic 3D human models with varied body shapes are fitted to each pose and rendered from 180 camera viewpoints with randomized clothing, backgrounds, and lighting conditions.
• Generative Adversarial Networks (GANs) are trained to minimize the distribution gap between real and synthetic RGB and depth images.
• Convolutional Neural Networks (CNNs) are trained to map input RGB and depth images to a shared, high-level invariant pose space, with features before the final Softmax layer used as pose representations.
• Temporal variations in the invariant pose features are modeled using a Fourier Temporal Pyramid to capture action dynamics.
• Linear SVMs are used for final action classification on the temporal feature sequences.

Experimental results

Research questions

• RQ1 Can synthetic data generation effectively model diverse variations in human poses, such as viewpoint, clothing, and lighting, to improve action recognition robustness?
• RQ2 To what extent can GANs bridge the domain gap between real and synthetic RGB-D images for training invariant pose models?
• RQ3 How well do CNN-based Human Pose Models generalize across different camera viewpoints and subjects when trained on synthetic data?
• RQ4 Does the combination of invariant pose features and Fourier Temporal Pyramid improve action recognition performance on cross-view and cross-subject benchmarks?
• RQ5 Can synthetic data significantly boost performance of existing models like C3D and LRCN even when pre-trained on large real datasets?

Key findings

• The proposed method outperforms existing state-of-the-art methods on three benchmark cross-view RGB-D action recognition datasets, achieving significant accuracy gains.
• On the UCF-101 Human Body Motion subset, the HPM model achieved 84.6% accuracy with twice the synthetic training data, approaching C3D’s 84.8% performance despite using only hundreds of thousands of frames versus millions of videos.
• The HPM model is nearly 7 times smaller than C3D, demonstrating high parameter efficiency with strong performance.
• The use of synthetic data improved performance of both C3D and LRCN even after pre-training on large-scale real datasets, indicating the value of controlled, diverse data augmentation.
• The Fourier Temporal Pyramid effectively models temporal dynamics of invariant pose features, contributing to robust action classification.
• The GAN-based refinement of synthetic images significantly improved the realism and distributional similarity to real data, enhancing downstream model generalization.