[Paper Review] A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms

Abdolrahim Kadkhodamohammadi, Afshin Gangi+2 more|arXiv (Cornell University)|Jan 25, 2017

Human Pose and Action Recognition37 references4 citations

TL;DR

This paper proposes a multi-view RGB-D approach for 3D human pose estimation in operating rooms, leveraging ConvNets for part detection and 3D pairwise constraints to enforce kinematic consistency. By fusing depth data across three synchronized cameras and using random forests to filter spurious detections, the method achieves significant accuracy gains over state-of-the-art methods, especially in multi-person, occluded, and visually cluttered OR environments.

ABSTRACT

Many approaches have been proposed for human pose estimation in single and multi-view RGB images. However, some environments, such as the operating room, are still very challenging for state-of-the-art RGB methods. In this paper, we propose an approach for multi-view 3D human pose estimation from RGB-D images and demonstrate the benefits of using the additional depth channel for pose refinement beyond its use for the generation of improved features. The proposed method permits the joint detection and estimation of the poses without knowing a priori the number of persons present in the scene. We evaluate this approach on a novel multi-view RGB-D dataset acquired during live surgeries and annotated with ground truth 3D poses.

Motivation & Objective

• To address the challenge of 3D human pose estimation in operating rooms, where standard RGB methods fail due to visual similarity of surfaces and textureless surgical gowns.
• To develop a system that detects and estimates 3D poses of multiple clinicians without prior knowledge of the number of people present.
• To leverage depth data not only for improved features but also for robust 3D point back-projection and multi-view correspondence, overcoming appearance-based matching failures.
• To jointly optimize 3D part positions using confidence maps, depth reprojection costs, and multi-view consistency, enhancing localization accuracy.

Proposed method

• A ConvNet-based part detector is trained on RGB-D data to detect body parts in each view, incorporating contextual information through large receptive fields.
• A 3D pairwise deformation model enforces kinematic constraints between body parts directly in 3D space, improving joint consistency in crowded scenes.
• Random forests are trained on scene-specific features to learn a prior about the operating room, filtering spurious skeleton candidates per view.
• Detections are back-projected into 3D using depth maps, enabling 3D correspondence even when points are visible in only one view.
• A novel energy function optimizes 3D part positions across views by combining detection confidence, depth reprojection costs, and multi-view evidence in four iterative steps.
• The optimization uses a coarse-to-fine search strategy with step sizes of 50 cm, 10 cm, 2 cm, and 1 cm to efficiently explore the 3D space.

Experimental results

Research questions

• RQ1 Can depth data improve 3D human pose estimation in operating rooms beyond enhancing appearance features?
• RQ2 How can multi-view RGB-D data be used to reliably detect and estimate multiple people’s poses without prior knowledge of person count?
• RQ3 To what extent does enforcing 3D kinematic constraints improve pose estimation in occluded and cluttered OR environments?
• RQ4 How effective is depth-based reprojection cost in guiding 3D pose optimization across multiple views?

Key findings

• The proposed method reduces 3D body part localization errors by up to 10 cm for skeletons with one or two supporting views and by ~3 cm for those with three views when using label voting, indicating improved detection reliability.
• The multi-view optimization reduces localization errors even for skeletons detected in only one view, demonstrating the effectiveness of depth-based reprojection costs and confidence fusion.
• Excluding the depth-based reprojection cost term leads to a significant performance drop, highlighting its critical role in accurate 3D pose refinement.
• The method achieves high-precision 3D skeleton detection after multi-view fusion, with the random forest prior significantly improving detection quality by filtering spurious candidates.
• The use of 3D pairwise constraints in the optimization function leads to more consistent and accurate body configurations, especially in multi-person and occluded scenarios.
• The proposed approach is the first multi-view RGB-D method for 3D human pose estimation that jointly detects and estimates poses without prior knowledge of the number of people in the scene.