[Paper Review] Exploring Severe Occlusion: Multi-Person 3D Pose Estimation with Gated Convolution

Renshu Gu, Gaoang Wang+1 more|arXiv (Cornell University)|Oct 31, 2020

Human Pose and Action Recognition35 references4 citations

TL;DR

This paper proposes a temporal gated convolution network for multi-person 3D human pose estimation under severe occlusion, leveraging motion capture-annotated data from a new moving-camera dataset (MMHuman). The method recovers missing 3D joints using gated convolutions and improves accuracy over SOTA methods, especially in occluded scenarios, by integrating temporal consistency and 3D-to-2D back-projection for global trajectory estimation.

ABSTRACT

3D human pose estimation (HPE) is crucial in many fields, such as human behavior analysis, augmented reality/virtual reality (AR/VR) applications, and self-driving industry. Videos that contain multiple potentially occluded people captured from freely moving monocular cameras are very common in real-world scenarios, while 3D HPE for such scenarios is quite challenging, partially because there is a lack of such data with accurate 3D ground truth labels in existing datasets. In this paper, we propose a temporal regression network with a gated convolution module to transform 2D joints to 3D and recover the missing occluded joints in the meantime. A simple yet effective localization approach is further conducted to transform the normalized pose to the global trajectory. To verify the effectiveness of our approach, we also collect a new moving camera multi-human (MMHuman) dataset that includes multiple people with heavy occlusion captured by moving cameras. The 3D ground truth joints are provided by accurate motion capture (MoCap) system. From the experiments on static-camera based Human3.6M data and our own collected moving-camera based data, we show that our proposed method outperforms most state-of-the-art 2D-to-3D pose estimation methods, especially for the scenarios with heavy occlusions.

Motivation & Objective

• Address the lack of real-world, moving-camera, multi-person 3D pose datasets with accurate ground truth, especially under heavy occlusion.
• Develop a robust 3D pose estimation method that can recover missing joints due to severe self- or inter-person occlusion.
• Improve generalization and performance on challenging, real-world video sequences where static-camera benchmarks fall short.
• Introduce a global trajectory estimation method that leverages temporal continuity and 3D-to-2D back-projection to localize poses in camera coordinates.
• Provide a new benchmark dataset (MMHuman) to support future research in occlusion-robust 3D human pose estimation under dynamic camera conditions.

Proposed method

• Proposes a temporal regression network with gated convolution modules to model long-range dependencies and recover missing 3D joints from 2D pose sequences.
• Uses a normalized 3D pose representation relative to the root joint, followed by a global trajectory recovery step via 3D-to-2D back-projection and temporal consistency.
• Applies gated convolutions inspired by image inpainting to selectively attend to visible joints and reconstruct occluded ones based on context.
• Trains the model on a combination of real-world occluded sequences and synthetic data to improve robustness to occlusion patterns.
• Employs a two-stage pipeline: 2D keypoint detection (via OpenPose) with confidence scores, followed by 2D-to-3D lifting using the gated temporal network.
• Introduces a novel loss function that combines joint-wise L2 loss and temporal smoothness regularization to enhance pose consistency across frames.

Experimental results

Research questions

• RQ1 Can a temporal gated convolution network effectively recover 3D poses from 2D joints in the presence of severe occlusion?
• RQ2 How does the proposed method compare to SOTA 2D-to-3D lifting models in terms of accuracy under heavy inter-person and self-occlusion?
• RQ3 To what extent does the global trajectory estimation method improve localization accuracy in moving-camera scenarios?
• RQ4 How generalizable is the model when fine-tuned on a new, real-world dataset with complex occlusions?
• RQ5 Can the proposed method outperform existing approaches on a newly collected, challenging dataset (MMHuman) with moving cameras and dense occlusions?

Key findings

• On the Human3.6M dataset, the proposed method achieves a 3D pose error of 43.8 mm at 50% occlusion, outperforming VideoPose3D (53.2 mm) and other SOTA methods.
• On the new MMHuman dataset, the proposed method achieves an average error of 78.3 mm, significantly outperforming Hossain et al. (118.5 mm), Pavllo et al. (84.2 mm), and Lin et al. (114.9 mm).
• Qualitative results show that the proposed method maintains natural and consistent pose estimations even during full-body occlusion, while VideoPose3D produces unnatural limb configurations.
• The method demonstrates robustness to occlusion, with a 12.5% relative improvement in error over VideoPose3D on the most challenging sequences (e.g., 'Kungfu' and 'PullUp').
• The global trajectory estimation method achieves accurate camera-coordinate localization without requiring additional supervision, outperforming VideoPose3D’s separate trajectory network in occluded settings.
• The model generalizes well to unseen actions in the MMHuman dataset without fine-tuning, indicating strong zero-shot generalization to real-world occlusion patterns.