[Paper Review] HumanMAC: Masked Motion Completion for Human Motion Prediction

Ling-Hao Chen, Jiawei Zhang+4 more|arXiv (Cornell University)|Feb 7, 2023

Human Pose and Action Recognition4 citations

TL;DR

HumanMAC proposes a novel masked motion completion framework for human motion prediction, leveraging a motion diffusion model trained to generate sequences from noise and adapted at inference via DCT-based denoising with masked conditioning on observed motions. The method achieves state-of-the-art performance with a single loss, end-to-end training, and superior motion diversity, including rare motion category switches, on CMU-MoCap and AMASS benchmarks.

ABSTRACT

Human motion prediction is a classical problem in computer vision and computer graphics, which has a wide range of practical applications. Previous effects achieve great empirical performance based on an encoding-decoding style. The methods of this style work by first encoding previous motions to latent representations and then decoding the latent representations into predicted motions. However, in practice, they are still unsatisfactory due to several issues, including complicated loss constraints, cumbersome training processes, and scarce switch of different categories of motions in prediction. In this paper, to address the above issues, we jump out of the foregoing style and propose a novel framework from a new perspective. Specifically, our framework works in a masked completion fashion. In the training stage, we learn a motion diffusion model that generates motions from random noise. In the inference stage, with a denoising procedure, we make motion prediction conditioning on observed motions to output more continuous and controllable predictions. The proposed framework enjoys promising algorithmic properties, which only needs one loss in optimization and is trained in an end-to-end manner. Additionally, it accomplishes the switch of different categories of motions effectively, which is significant in realistic tasks, e.g., the animation task. Comprehensive experiments on benchmarks confirm the superiority of the proposed framework. The project page is available at https://lhchen.top/Human-MAC.

Motivation & Objective

• To address limitations in existing encoding-decoding frameworks for human motion prediction, including complex loss functions, multi-stage training, and poor motion category switching.
• To enable controllable and diverse motion prediction by rethinking human motion prediction as a masked completion task.
• To simplify training with a single loss function and end-to-end optimization, avoiding hyperparameter tuning across multiple loss components.
• To achieve natural motion category transitions (e.g., walk to sit) even when not present in input sequences, enhancing realism and diversity.
• To develop a unified framework that models the full motion sequence holistically, ensuring continuity between observed and predicted frames.

Proposed method

• The framework trains a motion diffusion model to generate human motion sequences from random noise during the training phase.
• At inference, it treats motion prediction as a masked completion task by denoising a noisy representation of observed motions.
• A novel DCT-Completion mechanism combines denoised and noisy spectral representations using a learnable mask to condition predictions on observed motion.
• The method leverages Discrete Cosine Transform (DCT) to model low- and high-frequency components of motion, improving smoothness and accuracy.
• The entire pipeline is trained end-to-end with a single reconstruction loss, eliminating the need for multi-stage training or adversarial loss.
• The model is applied in an auto-regressive manner for long-horizon motion prediction by recursively conditioning on prior predictions.

Experimental results

Research questions

• RQ1 Can a masked motion completion framework outperform conventional encoding-decoding architectures in human motion prediction?
• RQ2 Can a single-loss, end-to-end diffusion model achieve state-of-the-art performance while simplifying training and hyperparameter tuning?
• RQ3 Can the model naturally generate motion category switches (e.g., from walking to sitting) even when such transitions are absent in the input?
• RQ4 How does DCT-based spectral modeling improve motion smoothness and accuracy compared to raw temporal modeling?
• RQ5 Can the framework generalize to long-horizon motion prediction with diverse, realistic outputs?

Key findings

• HumanMAC achieves SOTA performance on CMU-MoCap and AMASS datasets, with an APD of 8.021 and ADE of 0.433 on CMU-MoCap, outperforming prior VAE- and diffusion-based models.
• On AMASS, HumanMAC achieves an APD of 9.321 and ADE of 0.511, demonstrating strong generalization across diverse motion styles.
• The ablation study confirms that DCT modeling improves prediction accuracy and smoothness, reducing ADE from 0.444 (w/o DCT) to 0.369 (w/ DCT).
• HumanMAC enables diverse, continuous motion transitions—such as from squatting to standing or standing to walking—without requiring such transitions in the input sequence.
• The model generates more dynamic and less over-smoothed long-horizon predictions compared to baselines, which tend to produce static or repetitive motions.
• The framework uses only one loss function and is trained end-to-end, eliminating the need for multi-stage training or complex loss balancing.