[Paper Review] Cascaded Pyramid Network for 3D Human Pose Estimation Challenge

Sungeun Hong, Wonjin Jung+2 more|arXiv (Cornell University)|Oct 3, 2018

Human Pose and Action Recognition7 references3 citations

TL;DR

This paper proposes a two-stage 3D human pose estimation framework using a cascaded pyramid network (CPN) for robust 2D keypoint detection and a residual block-based 2D-to-3D lifting network. The method achieves a mean per joint position error (MPJPE) of 42.39 on the ECCV 2018 PoseTrack validation set, demonstrating strong performance on challenging monocular 3D pose estimation with improved handling of occluded joints.

ABSTRACT

Over the past decade, there has been a growing interest in human pose estimation. Although much work has been done on 2D pose estimation, 3D pose estimation has still been relatively studied less. In this paper, we propose a top-bottom based two-stage 3D estimation framework. GloabalNet and RefineNet in our 2D pose estimation process enable us to find occluded or invisible 2D joints while 2D-to-3D pose estimator composed of residual blocks is used to lift 2D joints to 3D joints effectively. The proposed method achieves promising results with mean per joint position error at 42.39 on the validation dataset on `3D Human Pose Estimation within the ECCV 2018 PoseTrack Challenge.'

Motivation & Objective

• Address the challenge of 3D human pose estimation from monocular RGB images, particularly in handling occluded or invisible 2D joints.
• Improve 3D pose estimation accuracy by leveraging a two-stage pipeline: first estimating 2D poses, then lifting them to 3D.
• Enhance robustness to scale variations between training (Human3.6M) and test (PoseTrack) datasets through a crop and resize (CR) module.
• Optimize the 2D-to-3D lifting network using residual blocks with batch normalization and max-norm constraints for stable training.
• Achieve state-of-the-art performance on the ECCV 2018 PoseTrack 3D Human Pose Estimation challenge.

Proposed method

• Use a top-bottom CPN architecture with GlobalNet and RefineNet to detect 2D keypoints, where GlobalNet uses feature pyramid networks and intermediate supervision to preserve spatial and semantic information.
• Apply RefineNet with online hard keypoint mining loss to improve estimation of occluded or invisible joints by aggregating multi-level features.
• Introduce a crop and resize (CR) module before and inverse crop and resize (ICR) after CPN inference to normalize scale differences between Human3.6M and the challenge dataset.
• Train a 2D-to-3D pose estimator using residual blocks with batch normalization and max-norm weight constraints to stabilize training and improve generalization.
• Normalize 2D and 3D poses by zero-centering around the hip joint and applying standardization to improve model convergence.
• Formulate the 2D-to-3D lifting as a minimization of the mean squared error loss over a batch of samples, using Eq. (1) as the objective function.

Experimental results

Research questions

• RQ1 Can a two-stage 3D pose estimation pipeline outperform one-stage methods in terms of accuracy and robustness on real-world video sequences?
• RQ2 How effective is the cascaded pyramid network (CPN) with GlobalNet and RefineNet in detecting occluded or invisible 2D keypoints?
• RQ3 To what extent does the crop and resize (CR) module improve performance when transferring a 2D pose model trained on Human3.6M to the PoseTrack dataset with different subject scales?
• RQ4 What is the optimal architecture and regularization strategy (e.g., batch normalization, max-norm) for the 2D-to-3D lifting network?
• RQ5 How does the combination of residual blocks, normalization, and weight constraints affect the final 3D pose estimation error?

Key findings

• The proposed method achieves a mean per joint position error (MPJPE) of 42.39 on the validation set of the ECCV 2018 PoseTrack 3D Human Pose Estimation challenge.
• The ablation study shows that using both batch normalization and max-norm constraints reduces MPJPE to 43.72, significantly outperforming configurations without these components.
• The inclusion of the crop and resize (CR) module improves performance by 17.5% compared to the baseline without CR, reducing MPJPE from 61.71 to 45.41 when combined with full regularization.
• A single residual block with 1024 dimensions achieves sufficient performance, indicating that the 2D-to-3D lifting head does not require deep architectures.
• The model demonstrates strong generalization on challenging cases, as shown in qualitative results where 3D predictions closely match ground truth even in complex poses.
• The use of RefineNet with online hard keypoint mining loss significantly improves detection of invisible or occluded joints, contributing to overall accuracy.