[Paper Review] High-Accuracy Real-Time Whole-Body Human Motion Tracking Based on Constrained Nonlinear Kalman Filtering

Jannik Steinbring, Christian Mandery+3 more|arXiv (Cornell University)|Nov 13, 2015

Human Pose and Action Recognition16 references3 citations

TL;DR

This paper proposes a real-time, high-accuracy whole-body human motion tracking system using constrained nonlinear Kalman filtering. By transforming joint limits into unconstrained periodic functions and applying the Smart Sampling Kalman Filter (S²KF), the method achieves robust tracking even with partial marker occlusions, outperforming gradient-based approaches and maintaining sub-8ms runtime on standard hardware while handling up to 16 missing markers on average.

ABSTRACT

We present a new online approach to track human whole-body motion from motion capture data, i.e., positions of labeled markers attached to the human body. Tracking in noisy data can be effectively performed with the aid of well-established recursive state estimation techniques. This allows us to systematically take noise of the marker measurements into account. However, as joint limits imposed by the human body have to be satisfied during estimation, first we transform this constrained estimation problem into an unconstrained one by using periodic functions. Then, we apply the Smart Sampling Kalman Filter to solve this unconstrained estimation problem. The proposed recursive state estimation approach makes the human motion tracking very robust to partial occlusion of markers and avoids any special treatment or reconstruction of the missed markers. A concrete implementation built on the kinematic human reference model of the Master Motor Map framework and a Vicon motion capture system is evaluated. Different captured motions show that our implementation can accurately estimate whole-body human motion in real-time and outperforms existing gradient-based approaches. In addition, we demonstrate its ability to smoothly handle incomplete marker data.

Motivation & Objective

• To enable real-time, high-accuracy whole-body human motion tracking from noisy, labeled marker data.
• To address the challenge of enforcing human joint limits in recursive state estimation.
• To develop a method that remains robust and accurate when markers are partially occluded or missing.
• To eliminate the need for special initialization (e.g., T-pose) or marker reconstruction in motion tracking.
• To demonstrate real-time performance and high accuracy on complex motions like handstands using a Vicon system and the Master Motor Map framework.

Proposed method

• Transform the constrained estimation problem—imposed by human joint limits—into an unconstrained one using periodic functions to represent joint angles.
• Apply the Smart Sampling Kalman Filter (S²KF) to recursively estimate the full human pose from noisy marker measurements.
• Use the kinematic reference model from the Master Motor Map (MMM) framework to define the human body’s structure and kinematics.
• Incorporate prior knowledge of marker positions and kinematic constraints into the state estimation process to improve accuracy.
• Leverage the recursive nature of the Kalman filter to maintain state estimates even when some markers are missing, without requiring marker reconstruction.
• Evaluate the method on a Vicon motion capture system with real-time processing on a standard CPU, ensuring low latency (<8ms).

Experimental results

Research questions

• RQ1 Can a constrained nonlinear Kalman filter be effectively used to track full-body human motion while respecting anatomical joint limits in real time?
• RQ2 How does the proposed method perform in terms of accuracy and robustness when markers are partially occluded or missing?
• RQ3 Can the use of periodic functions to represent joint angles enable accurate and unconstrained estimation within a Kalman filtering framework?
• RQ4 How does the proposed S²KF-based approach compare to existing gradient-based optimization methods in terms of accuracy and computational efficiency?
• RQ5 To what extent can the method maintain high tracking accuracy with incomplete marker data, such as when up to 50% of markers are hidden?

Key findings

• The proposed method achieves sub-8ms estimation runtime on an Intel Core i7-3770 CPU, enabling real-time whole-body motion tracking at 100Hz.
• The average marker distance between measured and estimated positions is below 5cm across all evaluated motions, with a maximum of 9cm during marker occlusion simulations.
• Even with up to 16 markers missing on average per frame (32% of 50 markers), the method maintains high accuracy, with marker distances increasing only slightly.
• The method outperforms existing gradient-based approaches, especially in complex motions like handstands, where the latter shows average marker distances over 60cm.
• The approach successfully handles partial occlusions without requiring marker reconstruction or special initialization, maintaining smooth and accurate pose estimation.
• Visual results in Fig. 6 show that estimated poses remain highly similar to ground truth even when more than 25 markers are hidden in some frames.