[Paper Review] Multi-level Motion Attention for Human Motion Prediction

Wei Mao, Miaomiao Liu+2 more|arXiv (Cornell University)|Jun 17, 2021

Human Pose and Action Recognition5 references4 citations

TL;DR

This paper proposes a multi-level motion attention mechanism for human motion prediction that captures temporal repetition by comparing current motion contexts with historical sub-sequences at joint, body part, and full-pose levels. By fusing these attention types and using a learnable graph convolutional network, the method achieves state-of-the-art performance on Human3.6M, AMASS, and 3DPW by effectively leveraging long-term motion patterns without recurrence or fixed temporal receptive fields.

ABSTRACT

Human motion prediction aims to forecast future human poses given a historical motion. Whether based on recurrent or feed-forward neural networks, existing learning based methods fail to model the observation that human motion tends to repeat itself, even for complex sports actions and cooking activities. Here, we introduce an attention based feed-forward network that explicitly leverages this observation. In particular, instead of modeling frame-wise attention via pose similarity, we propose to extract motion attention to capture the similarity between the current motion context and the historical motion sub-sequences. In this context, we study the use of different types of attention, computed at joint, body part, and full pose levels. Aggregating the relevant past motions and processing the result with a graph convolutional network allows us to effectively exploit motion patterns from the long-term history to predict the future poses. Our experiments on Human3.6M, AMASS and 3DPW validate the benefits of our approach for both periodical and non-periodical actions. Thanks to our attention model, it yields state-of-the-art results on all three datasets. Our code is available at https://github.com/wei-mao-2019/HisRepItself.

Motivation & Objective

• Address the limitation of RNN-based models in capturing long-term motion dependencies due to error accumulation and vanishing gradients.
• Overcome the shortcomings of frame-wise attention mechanisms that fail to capture motion direction and are sensitive to pose similarity across unrelated actions.
• Model motion repetitiveness not only in periodic actions (e.g., walking) but also in non-periodic, complex activities (e.g., cooking) by leveraging sub-sequence similarity.
• Develop a feed-forward architecture that dynamically attends to relevant historical motion patterns without relying on recurrent units or fixed temporal kernels.
• Improve generalization and performance on unseen datasets and longer prediction horizons by learning motion patterns from diverse historical sequences.

Proposed method

• Introduce motion attention that compares the current motion context (a sequence of recent frames) with historical motion sub-sequences, rather than individual frames.
• Define three levels of motion attention: joint-level (individual joints), body part-level (grouped joints), and full-pose-level (entire pose), each capturing different types of motion repetition.
• Use Discrete Cosine Transform (DCT) to encode temporal information in a non-recurrent, convolution-free manner, enabling long-range temporal modeling.
• Apply a learnable graph convolutional network (GCN) to model spatial relationships between joints, with the graph structure learned end-to-end instead of using a fixed kinematic tree.
• Fuse attention outputs from multiple levels using a learnable weighting mechanism to adaptively select the most relevant motion patterns for the current context.
• Train the model in a feed-forward fashion using mean squared error loss on predicted joint positions and angles, enabling inference without autoregressive generation.

Experimental results

Research questions

• RQ1 Can motion attention based on sub-sequence similarity improve long-term human motion prediction compared to frame-wise attention?
• RQ2 Does modeling motion repetitiveness at multiple levels (joint, body part, full pose) lead to better performance across diverse motion types?
• RQ3 How does the choice of temporal encoding (DCT vs. RNN or convolutional) affect prediction accuracy and generalization?
• RQ4 To what extent does learning the GCN adjacency matrix improve performance compared to using a fixed kinematic tree?
• RQ5 Can the model generalize to unseen datasets and longer prediction horizons without fine-tuning?

Key findings

• The proposed method achieves state-of-the-art performance on Human3.6M, AMASS, and 3DPW, outperforming prior SOTA methods including LTD (Mao et al., 2019) on all datasets.
• Fusing motion attention across all three levels (joint, part, pose) yields the best performance on joint position prediction, reducing average error by up to 33.5% at 1000ms horizon on Human3.6M.
• For joint angle prediction, fusing pose and part-level motion attention achieves the lowest error, with a 31.1% success rate in outperforming other attention types across sequences.
• The model generalizes well to unseen datasets without fine-tuning, maintaining strong performance on 3DPW when trained on Human3.6M and AMASS.
• Using a learnable GCN connectivity matrix reduces 3D error by up to 30% compared to using a pre-defined kinematic tree, demonstrating the importance of end-to-end structure learning.
• The ablation study confirms that DCT-based temporal encoding and learnable GCNs are both critical: removing either leads to significant performance drops, especially on long-horizon predictions.