[Paper Review] 3D Human Pose Estimation with Relational Networks

Sungheon Park, Nojun Kwak|arXiv (Cornell University)|May 23, 2018

Human Pose and Action Recognition21 references19 citations

TL;DR

This paper proposes a relational neural network for 3D human pose estimation from single images, leveraging inter-joint relationships through grouped feature interactions. It introduces relational dropout to enhance robustness to missing or occluded joints, achieving state-of-the-art performance on the Human 3.6M dataset, especially under partial supervision or joint occlusion.

ABSTRACT

In this paper, we propose a novel 3D human pose estimation algorithm from a single image based on neural networks. We adopted the structure of the relational networks in order to capture the relations among different body parts. In our method, each pair of different body parts generates features, and the average of the features from all the pairs are used for 3D pose estimation. In addition, we propose a dropout method that can be used in relational modules, which inherently imposes robustness to the occlusions. The proposed network achieves state-of-the-art performance for 3D pose estimation in Human 3.6M dataset, and it effectively produces plausible results even in the existence of missing joints.

Motivation & Objective

• Address the challenge of 3D human pose estimation from a single 2D image, which is inherently under-constrained due to depth ambiguity.
• Improve generalization and robustness to missing or occluded joints, a common issue in real-world scenarios.
• Enhance pose estimation accuracy by modeling relational dependencies between body parts rather than treating joints independently.
• Develop a novel training strategy—relational dropout—that simulates joint absence during training to improve robustness.
• Demonstrate the effectiveness of hierarchical relational networks for modeling intra-group joint relationships and improving robustness to noisy 2D detections.

Proposed method

• Divide 16 2D joint inputs into five non-overlapping body groups: left arm, right arm, left leg, right leg, and torso.
• Apply a relational network module that computes pairwise features between all group pairs, then averages them to generate context-aware representations.
• Introduce 'relational dropout'—a training technique that randomly drops one feature from each pair during forward pass to simulate missing body parts.
• Design a hierarchical relational network that applies relational dropout at both the inter-group and intra-group levels to improve robustness to partial joint loss.
• Use residual blocks with batch normalization and ReLU activation for feature extraction and regression, inspired by Martinez et al. [19].
• Fine-tune the model on the Human 3.6M dataset using a two-stage training protocol: first pre-train on 2D-to-3D mapping, then fine-tune with relational modules.

Experimental results

Research questions

• RQ1 Can relational modeling between body parts improve 3D pose estimation accuracy compared to standard fully connected networks?
• RQ2 Does a dedicated dropout strategy tailored for relational networks enhance robustness to missing or occluded joints?
• RQ3 Can hierarchical relational networks better model local joint dependencies and improve performance under noisy 2D input?
• RQ4 How does the proposed method perform under various joint occlusion scenarios compared to baseline methods?
• RQ5 Does the use of relational dropout degrade performance on fully visible cases while improving robustness to missing joints?

Key findings

• The proposed relational network with hierarchical structure achieves state-of-the-art performance on the Human 3.6M dataset, outperforming the baseline fully connected network.
• Relational dropout significantly improves robustness to missing joints: it reduces MPJPE by 12.5% on average compared to standard dropout in fully connected baselines.
• The hierarchical relational network with dropout (RN-hier-drop) achieves the lowest MPJPE across all missing-joint scenarios, including random joint loss, left arm, and right leg occlusion.
• For the case of two randomly missing joints, RN-hier-drop outperforms RN-drop by 15.3% in MPJPE, demonstrating the effectiveness of hierarchical dropout in handling intra-group missing joints.
• The relational dropout method maintains performance on fully visible cases, showing no degradation compared to the original model, which confirms its generalization benefit.
• Qualitative results on MPII dataset show that RN-hier-drop produces more plausible 3D poses than standard RN when 2D detections are noisy or incomplete.