[Paper Review] Dynamic Graph Reasoning for Multi-person 3D Pose Estimation

Zhongwei Qiu, Qiansheng Yang+2 more|arXiv (Cornell University)|Jul 22, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes GR-M3D, a novel bottom-up approach for multi-person 3D pose estimation that uses dynamic graph reasoning (DGR) to adaptively construct decoding graphs from multiple root keypoints and dense paths, enhancing robustness to occlusion and depth ambiguity. By integrating a scale and depth-aware refinement (SDAR) module, GR-M3D achieves state-of-the-art performance on three benchmarks, outperforming top-down and prior bottom-up methods with a 3.2% relative improvement in 3DPCK over baseline methods.

ABSTRACT

Multi-person 3D pose estimation is a challenging task because of occlusion and depth ambiguity, especially in the cases of crowd scenes. To solve these problems, most existing methods explore modeling body context cues by enhancing feature representation with graph neural networks or adding structural constraints. However, these methods are not robust for their single-root formulation that decoding 3D poses from a root node with a pre-defined graph. In this paper, we propose GR-M3D, which models the extbf{M}ulti-person extbf{3D} pose estimation with dynamic extbf{G}raph extbf{R}easoning. The decoding graph in GR-M3D is predicted instead of pre-defined. In particular, It firstly generates several data maps and enhances them with a scale and depth aware refinement module (SDAR). Then multiple root keypoints and dense decoding paths for each person are estimated from these data maps. Based on them, dynamic decoding graphs are built by assigning path weights to the decoding paths, while the path weights are inferred from those enhanced data maps. And this process is named dynamic graph reasoning (DGR). Finally, the 3D poses are decoded according to dynamic decoding graphs for each detected person. GR-M3D can adjust the structure of the decoding graph implicitly by adopting soft path weights according to input data, which makes the decoding graphs be adaptive to different input persons to the best extent and more capable of handling occlusion and depth ambiguity than previous methods. We empirically show that the proposed bottom-up approach even outperforms top-down methods and achieves state-of-the-art results on three 3D pose datasets.

Motivation & Objective

• To address the limitations of fixed-root decoding graphs in 3D multi-person pose estimation, especially under occlusion and depth ambiguity.
• To improve robustness in crowded scenes where body parts are frequently occluded and scale variations are high.
• To develop a dynamic, data-adaptive decoding mechanism that reasons over multiple potential keypoint paths rather than relying on a single root node.
• To achieve state-of-the-art performance in 3D pose estimation without relying on costly top-down detection stages.

Proposed method

• The method introduces a scale and depth-aware refinement (SDAR) module that enhances keypoint heatmaps, scale maps, depth maps, and 3D offset maps to improve feature representation.
• Multiple root keypoints and dense decoding paths are predicted from the enhanced feature maps, enabling multi-root-based reasoning.
• Dynamic decoding graphs are constructed by assigning learned soft weights to the predicted decoding paths, allowing adaptive graph structure per person.
• The dynamic graph reasoning (DGR) mechanism infers path weights from enhanced feature maps, enabling context-aware, adaptive message passing during 3D pose decoding.
• The framework uses a bottom-up paradigm, avoiding the need for human detection or instance association, and is compatible with various backbones including Hourglass, ResNet, and HRNet.
• The entire pipeline is end-to-end trainable and achieves real-time inference, even with heavy backbones like ResNet-101 and HRNet-32.

Experimental results

Research questions

• RQ1 Can a dynamic, data-adaptive decoding graph outperform fixed-structure graphs (e.g., star or tree) in handling occlusion and depth ambiguity in 3D multi-person pose estimation?
• RQ2 How does multi-root-based reasoning with learned path weights improve performance compared to single-root decoding strategies?
• RQ3 To what extent can a refinement module that jointly models scale and depth improve the quality of keypoint predictions and downstream decoding?
• RQ4 Does the proposed dynamic graph reasoning mechanism generalize to real-world, unconstrained, and crowded scenes?
• RQ5 Can a bottom-up approach with dynamic graph reasoning achieve state-of-the-art performance comparable to or better than top-down methods?

Key findings

• GR-M3D achieves 84.6 3DPCK on the MuPoTS-3D dataset using an Hourglass backbone, outperforming prior state-of-the-art methods.
• On the Human3.6M dataset, GR-M3D achieves a 5% relative improvement in MPJPE and a 16% improvement in PA MPJPE over the baseline using the same backbone.
• In crowded scenes with a crowd index over 0.7, GR-M3D shows a 5.9% relative improvement over the baseline, demonstrating strong robustness to occlusion.
• On the MuCo-3DHP dataset, GR-M3D achieves relative improvements of 2.5%, 6.5%, and 3.3% in $PCK_{rel}$, $PCK_{abs}$, and $AUC_{rel}$, respectively, over prior SOTA methods.
• GR-M3D maintains real-time inference speed (15–23 fps) even with heavy backbones like ResNet-101 and HRNet-32, with only a 2MB parameter increase over baseline.
• The model generalizes well to unconstrained, real-world images, including the COCO dataset, where it produces plausible 3D pose predictions without 3D annotations.