[Paper Review] 3D Pose Detection in Videos: Focusing on Occlusion

Justin Wang, Edward Xu+2 more|arXiv (Cornell University)|Jun 24, 2020

Human Pose and Action Recognition8 references4 citations

TL;DR

This paper proposes a two-stage 3D human pose estimation framework that improves accuracy for occluded joints in videos by integrating occlusion-aware temporal convolutional networks (TCNs) with 2D pose predictions from a stacked hourglass network. By introducing a refined cylinder man model for occlusion labeling and training the TCN to explicitly recognize occluded joints via a modified loss function, the method achieves a 5 mm lower mean per-joint position error (MPJPE) than a linear baseline and 0.1 mm lower than a TCN-only baseline, while reducing computational cost.

ABSTRACT

In this work, we build upon existing methods for occlusion-aware 3D pose detection in videos. We implement a two stage architecture that consists of the stacked hourglass network to produce 2D pose predictions, which are then inputted into a temporal convolutional network to produce 3D pose predictions. To facilitate prediction on poses with occluded joints, we introduce an intuitive generalization of the cylinder man model used to generate occlusion labels. We find that the occlusion-aware network is able to achieve a mean-per-joint-position error 5 mm less than our linear baseline model on the Human3.6M dataset. Compared to our temporal convolutional network baseline, we achieve a comparable mean-per-joint-position error of 0.1 mm less at reduced computational cost.

Motivation & Objective

• To improve 3D human pose estimation accuracy in videos, particularly for occluded joints that are difficult to detect due to self-occlusion or object obstruction.
• To address the limitations of existing 3D pose estimation models that fail to generalize well under occlusion by incorporating temporal context and explicit occlusion signals.
• To develop a more robust and efficient occlusion modeling strategy by refining the cylinder man model heuristic for generating occlusion labels from 3D ground truth keypoints.
• To train a temporal convolutional network (TCN) that leverages both 2D pose predictions and occlusion status to produce accurate 3D pose estimations, outperforming linear and standard TCN baselines.

Proposed method

• The method uses a stacked hourglass network to generate 2D keypoint predictions from video frames as input to the main 3D pose estimation pipeline.
• Occlusion labels are generated using a modified cylinder man model heuristic that maps 3D ground truth keypoints to 2D projections and determines whether a joint is occluded based on visibility in the camera view.
• The core model is a temporal convolutional network (TCN) that takes 2D joint coordinates and occlusion vectors as input, with the goal of predicting 3D joint positions across a sequence of frames.
• The TCN is trained with a modified loss function that explicitly incorporates ground truth occlusion labels, enabling the model to learn to reason about occluded joints more effectively.
• The framework is evaluated on the Human3.6M and HumanEva-I datasets, with ablation studies comparing different occlusion modeling strategies and network variants.
• A clustered occlusion heuristic is introduced as an alternative to the baseline Gaussian model, improving generalization and reducing overfitting to specific occlusion patterns.

Experimental results

Research questions

• RQ1 Can a temporal convolutional network (TCN) trained with explicit occlusion signals improve 3D pose estimation accuracy for occluded joints in video sequences compared to standard baselines?
• RQ2 How does the performance of the proposed occlusion-aware TCN compare to a linear regression baseline and a standard TCN baseline in terms of mean per-joint position error (MPJPE)?
• RQ3 To what extent does refining the cylinder man model heuristic for occlusion labeling enhance the model’s ability to predict 3D poses when joints are occluded?
• RQ4 Does training the TCN with ground truth occlusion labels in the loss function lead to better generalization and reduced error on occluded joints compared to models that do not use such supervision?
• RQ5 Can a simplified occlusion heuristic, such as the clustered occlusion model, achieve comparable or better performance than more complex heuristics while reducing computational overhead?

Key findings

• The occlusion-aware TCN achieved a mean per-joint position error (MPJPE) of 23.01 mm on the Human3.6M dataset, which is 5 mm lower than the linear baseline model.
• Compared to the standard TCN baseline, the proposed method achieved a 0.1 mm lower MPJPE while operating at reduced computational cost.
• The use of a refined cylinder man model for occlusion labeling significantly improved model performance, particularly on actions like 'SittingDown' where joint occlusion is prevalent.
• The clustered occlusion heuristic, which marks joints as occluded more liberally, outperformed the baseline Gaussian occlusion model and improved model robustness by encouraging generalization across different occlusion patterns.
• The model demonstrated strong performance on the 'SittingDown' action, where occlusion of the hip joint by the knees is common, due to effective use of temporal context and occlusion signals.
• Despite computational limitations, the model achieved a test error of 50.36 mm on Human3.6M when using occlusion vectors instead of heatmaps, indicating the viability of the approach even under constrained resources.