[Paper Review] Improving Human Motion Prediction Through Continual Learning

Mohammad Samin Yasar, Tariq Iqbal|arXiv (Cornell University)|Jul 1, 2021

Human Pose and Action Recognition37 references4 citations

TL;DR

This paper proposes a continual learning framework for human motion prediction that first pre-trains a robust representation on diverse benchmark data and then fine-tunes only the decoder on a target subject using curriculum learning. The approach achieves state-of-the-art performance across short-, mid-, and long-term horizons with minimal fine-tuning data, significantly outperforming baselines and mitigating catastrophic forgetting by preserving generalization while capturing individual motion subtleties.

ABSTRACT

Human motion prediction is an essential component for enabling closer human-robot collaboration. The task of accurately predicting human motion is non-trivial. It is compounded by the variability of human motion, both at a skeletal level due to the varying size of humans and at a motion level due to individual movement's idiosyncrasies. These variables make it challenging for learning algorithms to obtain a general representation that is robust to the diverse spatio-temporal patterns of human motion. In this work, we propose a modular sequence learning approach that allows end-to-end training while also having the flexibility of being fine-tuned. Our approach relies on the diversity of training samples to first learn a robust representation, which can then be fine-tuned in a continual learning setup to predict the motion of new subjects. We evaluated the proposed approach by comparing its performance against state-of-the-art baselines. The results suggest that our approach outperforms other methods over all the evaluated temporal horizons, using a small amount of data for fine-tuning. The improved performance of our approach opens up the possibility of using continual learning for personalized and reliable motion prediction.

Motivation & Objective

• To address the challenge of accurate human motion prediction in real-world human-robot collaboration scenarios where individual motion idiosyncrasies and distribution shifts hinder model generalization.
• To mitigate catastrophic forgetting in motion prediction models when adapting to new individuals with limited data.
• To enable personalized motion prediction by combining general representation learning with subject-specific fine-tuning in a continual learning setup.
• To improve prediction accuracy across short-, mid-, and long-term horizons by leveraging curriculum learning during fine-tuning.
• To demonstrate that fine-tuning only the decoder while keeping the encoder frozen can yield significant performance gains without overfitting.

Proposed method

• The framework uses a modular encoder-decoder architecture pre-trained on a diverse benchmark dataset using adversarial regularization to learn a robust general representation.
• In the second phase, only the decoder is fine-tuned on a specific subject’s data using a curriculum learning setup, where training starts with simpler sequences and progresses to more complex ones.
• The encoder weights are frozen during fine-tuning to preserve the general representation and reduce risk of catastrophic forgetting.
• The curriculum learning strategy involves training on one of four trials per subject and testing on the remaining three, with early stopping and validation-based model selection.
• The model is trained end-to-end in the first phase and then fine-tuned in a two-stage process: pre-training on benchmark data followed by subject-specific adaptation.
• Evaluation uses Mean Squared Error (MSE) as the primary metric, computed over joints and timesteps to assess pose prediction accuracy.

Experimental results

Research questions

• RQ1 Can a continual learning approach improve human motion prediction accuracy across diverse temporal horizons while minimizing catastrophic forgetting?
• RQ2 Does fine-tuning only the decoder on a small amount of subject-specific data lead to significant performance gains over non-curriculum learning baselines?
• RQ3 How does curriculum learning during fine-tuning affect the model’s ability to generalize to unseen motion patterns of a specific individual?
• RQ4 To what extent can a pre-trained general representation be effectively adapted to new subjects with minimal data using a modular, decoupled fine-tuning strategy?
• RQ5 Can the proposed method outperform the zero-velocity baseline and state-of-the-art models in both short- and long-term motion prediction?

Key findings

• The proposed method outperforms all baselines, including the state-of-the-art non-curriculum model and the zero-velocity baseline, across all evaluated temporal horizons (2, 4, 8, 10, 13, 15 frames).
• The performance gain is most significant in mid- and long-term prediction, where the model generates motion sequences perceptually closer to ground truth and with lower MSE.
• For subjects with high movement dynamics (e.g., Subjects 4 and 6), the model shows substantial improvement over the zero-velocity baseline, indicating effective learning of complex motion patterns.
• Even with only one trial used for fine-tuning, the model achieves lower MSE than non-curriculum learning baselines, demonstrating the effectiveness of the curriculum learning setup.
• The framework maintains robustness to catastrophic forgetting, as the encoder is not updated and the general representation remains stable across subject-specific adaptations.
• The results suggest that fine-tuning the decoder alone is sufficient for significant performance gains, with potential for further improvement through encoder fine-tuning in future work.