[Paper Review] Motion Prediction with Recurrent Neural Network Dynamical Models and Trajectory Optimization

Philipp Kratzer, Marc Toussaint+1 more|arXiv (Cornell University)|Jun 28, 2019

Autonomous Vehicle Technology and Safety11 references4 citations

TL;DR

This paper proposes a motion prediction framework that combines a GRU-based recurrent neural network for short-term human motion dynamics with gradient-based trajectory optimization to incorporate long-term goals and environmental constraints. By leveraging learned dynamics and optimizing trajectories toward target states, the method achieves significantly improved prediction accuracy over baselines, especially in long-horizon reaching motions.

ABSTRACT

Predicting human motion in unstructured and dynamic environments is difficult as humans naturally exhibit complex behaviors that can change drastically from one environment to the next. In order to alleviate this issue, we propose to encode the lower level aspects of human motion separately from the higher level geometrical aspects, which we believe will generalize better over environments. In contrast to our prior work~\cite{kratzer2018}, we encode the short-term behavior by using a state-of-the-art recurrent neural network structure instead of a Gaussian process. In order to perform longer-term behavior predictions that account for variation in tasks and environments, we propose to make use of gradient-based trajectory optimization. Preliminary experiments on real motion data demonstrate the efficacy of the approach.

Motivation & Objective

• Address the challenge of predicting complex human motion in unstructured, dynamic environments where behaviors vary significantly across settings.
• Overcome the scalability limitations of Gaussian Process-based models in motion prediction by replacing them with a deep learning alternative.
• Improve long-term motion prediction accuracy by integrating environmental and task-level constraints through trajectory optimization.
• Enable generalization across diverse human behaviors and environments by decoupling low-level dynamics learning from high-level goal specification.

Proposed method

• Use a sequence-to-sequence GRU network to model short-term human motion dynamics from raw motion capture data, trained to predict the next 0.5 seconds of full-body kinematics.
• Apply a sliding window approach to extract fixed-length 1-second trajectories from continuous motion data, with randomized base transformations to avoid position bias.
• Integrate the GRU-predicted dynamics into a trajectory optimization framework that minimizes a cost function incorporating goal-state constraints.
• Optimize the full trajectory using gradient-based optimization, ensuring the predicted path respects the target position while adhering to learned dynamics.
• Use a goal-conditioned cost function that penalizes deviation from the target hand position, enabling accurate long-horizon predictions.
• Retain the trained GRU model without retraining; only the optimization step is modified to include goal constraints, enabling plug-and-play integration with existing RNNs.

Experimental results

Research questions

• RQ1 Can a data-driven recurrent neural network replace Gaussian processes in motion prediction while maintaining generalization and scalability?
• RQ2 To what extent does incorporating goal-state constraints via trajectory optimization improve long-term motion prediction accuracy compared to raw RNN outputs?
• RQ3 How well can a GRU-based model generalize across diverse reaching tasks and object placements without task-specific fine-tuning?
• RQ4 Does trajectory optimization with a simple goal constraint significantly improve prediction quality when only the end-effector goal is provided?
• RQ5 Can the combination of learned dynamics and optimization handle complex full-body motion patterns in real-world human motion data?

Key findings

• The proposed method (GRU trajopt) reduces mean prediction error for full-body motion by 42% at 1000ms compared to the GRU baseline, achieving 1.19mm error versus 1.80mm.
• At the 1000ms prediction horizon, the wrist error of the proposed method drops to 0.00mm, achieving perfect convergence to the target position, outperforming linear interpolation and GRU baselines.
• The GRU trajopt method achieves a 1.15mm mean error for the wrist at 875ms, compared to 1.68mm for the GRU baseline, demonstrating improved long-term accuracy.
• The method maintains low error across all time steps, with a 25% reduction in wrist error at 500ms compared to the GRU baseline (0.87mm vs. 1.13mm).
• The use of trajectory optimization significantly improves prediction quality even with only the goal position of the right hand as input, indicating effective implicit modeling of human dynamics.
• The approach generalizes well across diverse reaching tasks involving objects at different heights, as demonstrated on a 5-minute test set with 25 distinct reaching motions.