[Paper Review] FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Cameras

Lan Xu, Lu Fang+5 more|arXiv (Cornell University)|Oct 29, 2016

Advanced Vision and Imaging32 references4 citations

TL;DR

FlyCap proposes a markerless motion capture system using three autonomous flying cameras (UAVs) with RGBD sensors to track full-body surface motions in large spaces. It employs a non-rigid surface registration method combined with visual odometry and a Gauss-Newton optimization to jointly estimate surface deformation and UAV poses, achieving accurate, real-time motion reconstruction without markers.

ABSTRACT

Aiming at automatic, convenient and non-instrusive motion capture, this paper presents a new generation markerless motion capture technique, the FlyCap system, to capture surface motions of moving characters using multiple autonomous flying cameras (autonomous unmanned aerial vehicles(UAV) each integrated with an RGBD video camera). During data capture, three cooperative flying cameras automatically track and follow the moving target who performs large scale motions in a wide space. We propose a novel non-rigid surface registration method to track and fuse the depth of the three flying cameras for surface motion tracking of the moving target, and simultaneously calculate the pose of each flying camera. We leverage the using of visual-odometry information provided by the UAV platform, and formulate the surface tracking problem in a non-linear objective function that can be linearized and effectively minimized through a Gaussian-Newton method. Quantitative and qualitative experimental results demonstrate the competent and plausible surface and motion reconstruction results

Motivation & Objective

• To enable automatic, non-invasive, and large-scale motion capture without markers.
• To address the challenge of tracking complex, non-rigid surface deformations during full-body motion in wide environments.
• To develop a system that simultaneously estimates the 3D pose of multiple flying cameras and the surface motion of a moving subject.
• To leverage visual odometry and depth data from UAVs for robust, real-time surface reconstruction.

Proposed method

• The system uses three autonomous UAVs each equipped with an RGBD camera to capture depth and color data during motion.
• A non-rigid surface registration method fuses depth data from multiple cameras to track the deforming surface of a moving subject.
• The surface tracking problem is formulated as a non-linear optimization objective function that incorporates visual odometry and depth measurements.
• The optimization is linearized and solved using a Gauss-Newton method to jointly estimate surface shape and UAV poses.
• The method integrates visual odometry data from the UAV platforms to improve pose estimation accuracy.
• The system enables real-time, markerless 3D reconstruction of human motion in unconstrained, large-scale environments.

Experimental results

Research questions

• RQ1 Can multiple autonomous flying cameras achieve accurate and robust markerless motion capture in large, unconstrained spaces?
• RQ2 How can depth data from multiple moving cameras be effectively fused to reconstruct non-rigid surface deformations?
• RQ3 Can visual odometry from UAVs be reliably integrated into a non-rigid surface tracking framework to improve pose estimation?
• RQ4 What is the performance of the system in terms of reconstruction accuracy and real-time capability?

Key findings

• The FlyCap system achieves accurate 3D reconstruction of non-rigid surface motions during large-scale human movements.
• Joint optimization of surface shape and UAV poses using Gauss-Newton minimization improves tracking stability and accuracy.
• The system demonstrates real-time performance suitable for practical motion capture applications.
• The use of visual odometry enhances UAV pose estimation, reducing drift and improving fusion quality.
• Qualitative results show plausible and detailed surface reconstructions across diverse motion sequences.
• Quantitative results confirm the system's competence in capturing complex, full-body motions without markers.