[Paper Review] HMOR: Hierarchical Multi-Person Ordinal Relations for Monocular Multi-Person 3D Pose Estimation

Jiefeng Li, Can Wang+3 more|arXiv (Cornell University)|Aug 1, 2020

Human Pose and Action Recognition74 references7 citations

TL;DR

This paper proposes HMOR (Hierarchical Multi-Person Ordinal Relations), a novel supervision method that encodes depth and angular relationships among persons, body parts, and joints in a hierarchical manner to improve global consistency in monocular multi-person 3D pose estimation. The integrated end-to-end top-down model jointly estimates human detection, depth, and 3D poses using a coarse-to-fine depth refinement strategy, achieving state-of-the-art performance with lower computation and parameter costs on MuPoTS-3D and CMU Panoptic datasets.

ABSTRACT

Remarkable progress has been made in 3D human pose estimation from a monocular RGB camera. However, only a few studies explored 3D multi-person cases. In this paper, we attempt to address the lack of a global perspective of the top-down approaches by introducing a novel form of supervision - Hierarchical Multi-person Ordinal Relations (HMOR). The HMOR encodes interaction information as the ordinal relations of depths and angles hierarchically, which captures the body-part and joint level semantic and maintains global consistency at the same time. In our approach, an integrated top-down model is designed to leverage these ordinal relations in the learning process. The integrated model estimates human bounding boxes, human depths, and root-relative 3D poses simultaneously, with a coarse-to-fine architecture to improve the accuracy of depth estimation. The proposed method significantly outperforms state-of-the-art methods on publicly available multi-person 3D pose datasets. In addition to superior performance, our method costs lower computation complexity and fewer model parameters.

Motivation & Objective

• To address the lack of global scene understanding in top-down monocular multi-person 3D pose estimation methods.
• To reduce depth ambiguity and occlusion challenges in complex, cluttered scenes with multiple interacting persons.
• To improve the accuracy of absolute 3D pose estimation by incorporating hierarchical ordinal relationships between persons, body parts, and joints.
• To develop an end-to-end trainable, integrated model that jointly performs human detection, depth estimation, and 3D pose regression.
• To achieve superior performance with reduced model complexity and computation cost compared to existing methods.

Proposed method

• Introduces Hierarchical Multi-person Ordinal Relations (HMOR), which models depth and angular relations at three levels: person-instance, body part, and joint.
• Encodes ordinal supervision as relative depth and angle constraints across hierarchical levels to enforce global consistency in predicted 3D poses.
• Proposes an integrated top-down model that performs human detection, 3D pose estimation, and human depth estimation simultaneously in an end-to-end manner.
• Employs a coarse-to-fine depth estimation architecture: first generates a global depth map, then refines individual human depths via residual correction.
• Utilizes a 2D-3D mixed training strategy compatible with 2D pose annotations, enabling efficient and effective pretraining.
• Applies a novel loss function that combines HMOR supervision with standard 3D pose and depth regression losses for joint optimization.

Experimental results

Research questions

• RQ1 Can hierarchical ordinal relations between persons, body parts, and joints improve global consistency in monocular multi-person 3D pose estimation?
• RQ2 How does HMOR supervision compare to direct regression of absolute 3D poses in terms of accuracy and robustness?
• RQ3 To what extent does a coarse-to-fine depth estimation strategy improve the accuracy of absolute 3D pose estimation from a single RGB image?
• RQ4 Can an integrated end-to-end model jointly optimize detection, depth, and 3D pose estimation while reducing computational cost?
• RQ5 How effective is the proposed method when trained with mixed 2D and 3D annotations, and how does it generalize to in-the-wild scenarios?

Key findings

• The proposed method achieves a 12.3 PCK abs improvement over state-of-the-art methods on the MuPoTS-3D dataset.
• On the CMU Panoptic dataset, the method improves MPJPE by 20.5 mm compared to prior work, demonstrating superior absolute pose estimation accuracy.
• The model reduces model parameters by 73% and GFLOPs by 41% compared to Moon et al. [39], the only open-source baseline for multi-person 3D pose estimation.
• HMOR supervision leads to a 7.4 mm MPJPE improvement over direct absolute 3D pose regression, with instance-level relations contributing most to absolute pose accuracy.
• The coarse-to-fine depth refinement strategy reduces MPJPE by 15.1 mm compared to direct depth output, proving its necessity for accurate depth estimation.
• Sampling additional view directions during training improves MPJPE by 0.6 mm, indicating enhanced generalization to complex scenes.