[Paper Review] Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation

Qihao Liu, Yi Zhang+2 more|arXiv (Cornell University)|Jul 29, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes HUPOR, a novel framework for multi-person 3D human pose estimation that explicitly reasons about occluded joints using visible cues. By decoupling visible keypoint detection from occluded keypoint reasoning via a Deeply Supervised Encoder Distillation (DSED) network and generating accurate pseudo-occlusion labels using a Skeleton-guided Shape Fitting (SSF) method, the approach achieves state-of-the-art performance on multiple benchmarks, significantly improving robustness under heavy occlusion.

ABSTRACT

Occlusion poses a great threat to monocular multi-person 3D human pose estimation due to large variability in terms of the shape, appearance, and position of occluders. While existing methods try to handle occlusion with pose priors/constraints, data augmentation, or implicit reasoning, they still fail to generalize to unseen poses or occlusion cases and may make large mistakes when multiple people are present. Inspired by the remarkable ability of humans to infer occluded joints from visible cues, we develop a method to explicitly model this process that significantly improves bottom-up multi-person human pose estimation with or without occlusions. First, we split the task into two subtasks: visible keypoints detection and occluded keypoints reasoning, and propose a Deeply Supervised Encoder Distillation (DSED) network to solve the second one. To train our model, we propose a Skeleton-guided human Shape Fitting (SSF) approach to generate pseudo occlusion labels on the existing datasets, enabling explicit occlusion reasoning. Experiments show that explicitly learning from occlusions improves human pose estimation. In addition, exploiting feature-level information of visible joints allows us to reason about occluded joints more accurately. Our method outperforms both the state-of-the-art top-down and bottom-up methods on several benchmarks.

Motivation & Objective

• To address the persistent challenge of occlusion in monocular multi-person 3D human pose estimation, which severely degrades performance in real-world scenarios.
• To overcome the limitations of existing methods that rely on implicit reasoning or pose priors that generalize poorly to unseen occlusion patterns.
• To enable explicit occlusion reasoning by modeling the human-like ability to infer occluded joints from visible joints and contextual cues.
• To generate accurate pseudo-occlusion labels for training, overcoming the lack of per-joint visibility annotations in existing datasets.
• To improve both 2D and 3D human pose estimation by integrating a reasoning module that enhances detection and grouping robustness.

Proposed method

• The method splits 3D human pose estimation into two stages: (1) visible keypoint detection using a state-of-the-art bottom-up detector, and (2) occluded keypoint reasoning using a dedicated module.
• A Deeply Supervised Encoder Distillation (DSED) network is proposed to explicitly infer occluded joints by leveraging feature-level representations from visible joints and contextual information.
• A Skeleton-guided human Shape Fitting (SSF) method is introduced to generate accurate pseudo-occlusion labels by fitting parametric SMPL models to images and using projection-based visibility checks.
• The SSF method improves upon existing shape fitting by combining 2D/3D pose-based and image-based fitting, enabling detection of self-occlusion, object occlusion, and mutual occlusion between people.
• The reasoning module is trained using the generated pseudo-occlusion labels, allowing the network to learn explicit occlusion reasoning without requiring manual annotations.
• The framework is compatible with both 2D and 3D pose estimation by modifying the reasoning module’s depth-aware components.

Experimental results

Research questions

• RQ1 Can explicit occlusion reasoning significantly improve the robustness of multi-person 3D human pose estimation under heavy occlusion?
• RQ2 How can accurate pseudo-occlusion labels be generated for training when ground-truth visibility annotations are absent?
• RQ3 Can a dedicated reasoning module that leverages visible joint features outperform end-to-end or implicit reasoning approaches?
• RQ4 To what extent does the DSED network improve occluded keypoint prediction compared to standard hourglass or transformer-based architectures?
• RQ5 Does the proposed method generalize to 2D human pose estimation and improve performance without depth supervision?

Key findings

• HUPOR achieves state-of-the-art performance on the COCO test-dev 2017 benchmark, improving AP from 68.4 to 69.5 with the DSED reasoning module.
• On the 3DPW dataset, the method reduces MPJPE from 80.0 to 79.1 and PA-MPJPE from 48.8 to 49.3, demonstrating improved mesh recovery accuracy.
• The SSF method generates occlusion labels with 88.5% precision and 80.5% recall, significantly outperforming baseline methods like Cylinder (65.3% precision) and HybrIK (80.0% precision).
• Adding the reasoning module improves 2D pose estimation by up to 1.7 AP on COCO, showing generalization beyond 3D tasks.
• The DSED network is crucial: removing it reduces performance by over 2 AP on COCO, confirming its role in effective feature distillation for occlusion reasoning.
• The method reduces common failure cases such as missing persons, extra person detections, and incomplete skeletons under occlusion, as shown in qualitative comparisons.