[Paper Review] Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo

Jiahao Lin, Gim Hee Lee|arXiv (Cornell University)|Apr 6, 2021

Human Pose and Action Recognition23 references4 citations

TL;DR

This paper proposes a plane-sweep stereo-based method for multi-view multi-person 3D pose estimation that regresses joint depths directly without explicit cross-view correspondence matching. By leveraging geometric consistency across views through plane sweeping and a coarse-to-fine depth regression scheme, the approach achieves state-of-the-art accuracy while being up to 20x faster than prior learning-based methods like VoxelPose.

ABSTRACT

Existing approaches for multi-view multi-person 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views and solve for the 3D pose estimation for each person. Establishing cross-view correspondences is challenging in multi-person scenes, and incorrect correspondences will lead to sub-optimal performance for the multi-stage pipeline. In this work, we present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot. Specifically, we propose to perform depth regression for each joint of each 2D pose in a target camera view. Cross-view consistency constraints are implicitly enforced by multiple reference camera views via the plane sweep algorithm to facilitate accurate depth regression. We adopt a coarse-to-fine scheme to first regress the person-level depth followed by a per-person joint-level relative depth estimation. 3D poses are obtained from a simple back-projection given the estimated depths. We evaluate our approach on benchmark datasets where it outperforms previous state-of-the-arts while being remarkably efficient. Our code is available at https://github.com/jiahaoLjh/PlaneSweepPose.

Motivation & Objective

• To address the challenge of cross-view correspondence matching in multi-view multi-person 3D pose estimation, which is error-prone and degrades performance in multi-stage pipelines.
• To enable end-to-end, single-shot 3D pose estimation by jointly fusing multi-view information through geometric consistency instead of explicit matching.
• To improve computational efficiency compared to 3D voxel-based methods like VoxelPose, which incur heavy 3D convolution costs on sparse scenes.
• To generalize across different camera configurations without requiring prior knowledge of the 3D scene volume or camera layout.
• To achieve high accuracy in depth regression for each 2D pose keypoint using a coarse-to-fine scheme with implicit multi-view consistency.

Proposed method

• The method performs depth regression for each 2D pose joint in a target camera view using plane sweep stereo, where virtual depth planes are back-projected and warped to reference views.
• Cross-view consistency is measured at each depth level by comparing warped features across views, which guides the depth regression network.
• A two-stage coarse-to-fine depth estimation is employed: first estimating person-level depth, then joint-level relative depth within a narrow range.
• The depth predictions are used to back-project 2D poses into 3D space, forming 3D poses that are merged via distance-based clustering across views.
• The framework uses 1D convolutions per 2D pose instead of 3D convolutions over full volumetric space, significantly reducing computation.
• The entire pipeline is trained end-to-end and does not require explicit 2D pose matching or triangulation.

Experimental results

Research questions

• RQ1 Can plane sweep stereo be effectively adapted to perform joint-level depth regression for 2D pose detections in multi-view multi-person 3D pose estimation?
• RQ2 Can a coarse-to-fine depth estimation scheme improve accuracy while maintaining computational efficiency?
• RQ3 Does implicit multi-view consistency via plane sweeping outperform explicit cross-view correspondence matching in complex scenes?
• RQ4 How does the method generalize across different camera configurations without prior knowledge of the 3D scene layout?
• RQ5 Can the method achieve state-of-the-art performance with significantly lower inference cost than 3D voxel-based approaches?

Key findings

• The method achieves 97.0% average accuracy on the Campus dataset, improving from 96.7% in the prior state-of-the-art.
• On the Shelf dataset, the method reaches 97.9% average accuracy, up from 97.4% in previous methods.
• On the CMU Panoptic dataset, the method reduces MPJPE to 16.75mm, a 0.93mm improvement over VoxelPose’s 17.68mm.
• The method achieves up to 20x faster inference than VoxelPose, with a frame rate suitable for real-time applications.
• Even with only two cameras, the method maintains over 92% accuracy, demonstrating robustness in low-visibility settings.
• The method generalizes well across camera configurations, performing comparably when trained on random camera sets and tested on fixed ones.