[Paper Review] 3D human pose estimation in video with temporal convolutions and semi-supervised training

Dario Pavllo, Christoph Feichtenhofer+2 more|arXiv (Cornell University)|Nov 28, 2018

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a fully convolutional 3D human pose estimation model using dilated temporal convolutions on 2D keypoint trajectories, achieving a 6 mm reduction in mean per-joint position error (11% relative improvement) on Human3.6M compared to prior work. It further introduces back-projection, a semi-supervised training method that leverages unlabeled video by predicting 3D poses and back-projecting to 2D, reducing error by up to 14.7 mm MPJPE when labeled data is scarce.

ABSTRACT

In this work, we demonstrate that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints. We also introduce back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data. We start with predicted 2D keypoints for unlabeled video, then estimate 3D poses and finally back-project to the input 2D keypoints. In the supervised setting, our fully-convolutional model outperforms the previous best result from the literature by 6 mm mean per-joint position error on Human3.6M, corresponding to an error reduction of 11%, and the model also shows significant improvements on HumanEva-I. Moreover, experiments with back-projection show that it comfortably outperforms previous state-of-the-art results in semi-supervised settings where labeled data is scarce. Code and models are available at https://github.com/facebookresearch/VideoPose3D

Motivation & Objective

• To improve 3D human pose estimation in video by leveraging temporal context more effectively than recurrent models.
• To address the scarcity of labeled 3D pose data by introducing a semi-supervised method that uses unlabeled video without requiring 2D ground truth or multi-view data.
• To develop a model that is both accurate and efficient, enabling parallel processing and reduced computational complexity compared to RNN-based approaches.
• To demonstrate that back-projection, a cycle-consistency-inspired method, effectively improves generalization when labeled data is limited.

Proposed method

• Uses a fully convolutional architecture with dilated temporal convolutions to model long-range dependencies in 2D keypoint trajectories.
• Applies dilated convolutions to achieve a large receptive field with low computational cost, enabling efficient modeling of long sequences.
• Introduces back-projection: a semi-supervised training scheme that predicts 3D poses from 2D keypoints and projects them back to 2D to form a consistency loss.
• Incorporates a bone length regularization term to enforce kinematic consistency and improve pose validity during back-projection.
• Trains the model using a combination of supervised loss on labeled data and consistency loss from unlabeled video via back-projection.
• Uses only intrinsic camera parameters in the back-projection step, avoiding the need for extrinsic parameters or 2D ground truth in unlabeled data.

Experimental results

Research questions

• RQ1 Can a fully convolutional model with dilated temporal convolutions outperform RNN-based models in 3D human pose estimation on video?
• RQ2 Can back-projection, a semi-supervised method based on cycle consistency, effectively improve 3D pose estimation when labeled data is limited?
• RQ3 How does the performance of the proposed method scale with decreasing amounts of labeled data, especially in low-resource settings?
• RQ4 To what extent does enforcing kinematic constraints via bone length regularization improve generalization in semi-supervised training?

Key findings

• The proposed fully convolutional model achieves a 6 mm reduction in mean per-joint position error (from 53.1 mm to 47.1 mm) on the Human3.6M dataset, representing an 11% relative improvement over the previous state of the art.
• The model outperforms prior methods on HumanEva-I, demonstrating generalization beyond the Human3.6M benchmark.
• In semi-supervised settings with only 5,000 or fewer labeled frames, the back-projection method reduces error by up to 14.7 mm MPJPE compared to a strong supervised baseline.
• When trained on ground-truth 2D poses, the semi-supervised method achieves up to 22.6 mm MPJPE improvement over the supervised baseline on 1% of S1, highlighting the potential of better 2D keypoint detectors.
• Removing the bone length regularization term in back-projection increases error from 78.1 mm to 91.3 mm on 1% of S1, demonstrating its critical role in maintaining pose validity.
• The model achieves ~150k FPS inference speed on a single GPU, with speed independent of batch size due to parallel temporal processing, outperforming RNN-based models in efficiency.