[Paper Review] Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time

Yinghao Huang, Manuel Kaufmann+4 more|arXiv (Cornell University)|Oct 10, 2018

Human Pose and Action RecognitionComputer Science60 references58 citations

TL;DR

DIP uses a deep recurrent network to reconstruct full-body SMPL poses from only 6 IMUs in real time, by synthesizing training data and leveraging bidirectional temporal modeling with a real-time sliding window.

ABSTRACT

We demonstrate a novel deep neural network capable of reconstructing human full body pose in real-time from 6 Inertial Measurement Units (IMUs) worn on the user's body. In doing so, we address several difficult challenges. First, the problem is severely under-constrained as multiple pose parameters produce the same IMU orientations. Second, capturing IMU data in conjunction with ground-truth poses is expensive and difficult to do in many target application scenarios (e.g., outdoors). Third, modeling temporal dependencies through non-linear optimization has proven effective in prior work but makes real-time prediction infeasible. To address this important limitation, we learn the temporal pose priors using deep learning. To learn from sufficient data, we synthesize IMU data from motion capture datasets. A bi-directional RNN architecture leverages past and future information that is available at training time. At test time, we deploy the network in a sliding window fashion, retaining real time capabilities. To evaluate our method, we recorded DIP-IMU, a dataset consisting of $10$ subjects wearing 17 IMUs for validation in $64$ sequences with $330\,000$ time instants; this constitutes the largest IMU dataset publicly available. We quantitatively evaluate our approach on multiple datasets and show results from a real-time implementation. DIP-IMU and the code are available for research purposes.

Motivation & Objective

• Demonstrate real-time full-body pose reconstruction from six IMUs without full camera setups.
• Leverage synthetic IMU data generated from large Mocap datasets to train a deep model.
• Model temporal dynamics with recurrent neural networks to improve pose estimation under sparse sensing.
• Incorporate acceleration reconstruction and normalization to improve generalization to real data.
• Provide a publicly available DIP-IMU dataset and code for research use.

Proposed method

• Synthesize IMU data by placing virtual sensors on the SMPL mesh and deriving orientations via forward kinematics and accelerations via finite differences.
• Train a bidirectional LSTM-based RNN to map IMU inputs to SMPL pose parameters, using a log-likelihood loss.
• Use an auxiliary task to reconstruct input accelerations, ensuring acceleration information propagates through the network.
• Model target poses with a Normal distribution during training and use the corresponding mean and variance in the loss.
• Normalize IMU inputs by root-relative frame to achieve orientation invariance to user heading.
• Fine-tune pre-trained models on real IMU data to bridge the gap between synthetic and real distributions.

Experimental results

Research questions

• RQ1 Can six IMUs provide accurate full-body SMPL poses in real time?
• RQ2 How can synthetic IMU data from Mocap datasets be leveraged to train a robust pose estimator?
• RQ3 Does a bidirectional temporal model improve pose accuracy for sparse sensor input compared to unidirectional models?
• RQ4 What training strategies (acceleration reconstruction, normalization, regularization) improve real-world generalization?
• RQ5 What is the impact of using a sliding window at test time on latency and accuracy?

Key findings

• The method achieves real-time full-body pose estimation from 6 IMUs.
• Bi-directional temporal modeling with a sliding window yields improved accuracy with modest latency (about 85 ms).
• An auxiliary acceleration reconstruction task improves pose predictions.
• Training on synthetic IMU data, followed by fine-tuning on real data, narrows the gap between synthetic and real measurements.
• DIP achieves an angular error of 15.85 degrees on TotalCapture and DIP-IMU benchmarks, outperforming a competing offline approach while running in real time.