[Paper Review] Markerless 3D human pose tracking through multiple cameras and AI: Enabling high accuracy, robustness, and real-time performance

Luca Fortini, Mattia Leonori+3 more|arXiv (Cornell University)|Mar 31, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a markerless 3D human pose tracking framework using multi-camera setups and AI-based 2D pose estimation, fusing data via a weighted least squares (WLS) triangulation method to achieve high accuracy, real-time performance, and robustness in unconstrained environments. The approach enables real-world deployment with minimal user interference and high reliability across diverse applications.

ABSTRACT

Tracking 3D human motion in real-time is crucial for numerous applications across many fields. Traditional approaches involve attaching artificial fiducial objects or sensors to the body, limiting their usability and comfort-of-use and consequently narrowing their application fields. Recent advances in Artificial Intelligence (AI) have allowed for markerless solutions. However, most of these methods operate in 2D, while those providing 3D solutions compromise accuracy and real-time performance. To address this challenge and unlock the potential of visual pose estimation methods in real-world scenarios, we propose a markerless framework that combines multi-camera views and 2D AI-based pose estimation methods to track 3D human motion. Our approach integrates a Weighted Least Square (WLS) algorithm that computes 3D human motion from multiple 2D pose estimations provided by an AI-driven method. The method is integrated within the Open-VICO framework allowing simulation and real-world execution. Several experiments have been conducted, which have shown high accuracy and real-time performance, demonstrating the high level of readiness for real-world applications and the potential to revolutionize human motion capture.

Motivation & Objective

• Address the limitations of marker-based motion capture systems, including user discomfort, setup complexity, and restricted usability in real-world scenarios.
• Overcome the trade-off between accuracy, real-time performance, and robustness in existing markerless 3D pose estimation methods.
• Enable practical, real-time 3D human motion tracking using only standard RGB cameras and AI-based 2D pose estimators, eliminating the need for physical markers.
• Develop a triangulation framework that fuses multi-view 2D keypoint detections with adaptive weighting to improve 3D reconstruction accuracy under occlusions and low-confidence predictions.
• Integrate the system into the Open-VICO framework to support both simulation and real-world deployment across diverse environments.

Proposed method

• Utilizes multiple RGB cameras to capture synchronized video streams of human motion, avoiding line-of-sight issues through spatial diversity.
• Employs AI-based 2D pose estimation (e.g., OpenPose) to detect 21 body joints in each camera view, using confidence scores and visibility metrics as input.
• Applies a Weighted Least Squares (WLS) triangulation method to reconstruct 3D joint positions from 2D detections, minimizing weighted residuals.
• Computes adaptive weights per joint and camera using three components: confidence score ($w_{c,j}^s$), distance-based reliability ($w_{c,j}^d$), and orthogonality index ($w_{c,j}^o$) for visibility.
• Implements a thresholding strategy: joints with confidence score below 0.4 are excluded from triangulation to reduce noise and outliers.
• Applies the International Society of Biomechanics (ISB) standard for Joint Coordinate System (JCS) to compute 3D segment angles and anatomical frames from reconstructed 3D joints.

Experimental results

Research questions

• RQ1 Can multi-view 2D pose estimation combined with WLS triangulation achieve real-time, high-accuracy 3D human pose reconstruction without markers?
• RQ2 How do adaptive weights based on confidence, distance, and visibility improve 3D pose accuracy under occlusions and low-quality detections?
• RQ3 To what extent does the proposed framework outperform existing markerless 3D pose estimation methods in terms of accuracy and latency in real-world settings?
• RQ4 Can the system maintain robust performance across diverse environments and complex human movements without requiring specialized calibration or controlled backgrounds?
• RQ5 How effective is the integration of the framework within the Open-VICO platform for both simulation and real-world deployment?

Key findings

• The proposed framework achieves high accuracy in 3D human pose reconstruction, with results demonstrating strong performance comparable to marker-based systems in unconstrained environments.
• The use of adaptive weights—based on confidence, distance, and visibility—significantly improves robustness, especially in the presence of occlusions and low-confidence detections.
• The system operates in real time, with low latency suitable for interactive applications such as VR, robotics, and digital twins.
• Excluding low-confidence joints (confidence < 0.4) through a thresholding mechanism enhances overall triangulation accuracy by reducing the influence of erroneous 2D detections.
• The integration within the Open-VICO framework enables seamless simulation and real-world deployment, validating the system’s practical readiness for diverse applications.
• The method demonstrates strong generalization across different body poses and camera configurations, maintaining consistent performance without requiring retraining or complex calibration.