[Paper Review] Single-Shot Motion Completion with Transformer

Yinglin Duan, Tianyang Shi+5 more|arXiv (Cornell University)|Mar 1, 2021

Video Analysis and Summarization36 references29 citations

TL;DR

A transformer-based, non-autoregressive model completes missing motion frames in a unified framework for in-betweening, in-filling, and blending, achieving state-of-the-art accuracy on LaFAN1.

ABSTRACT

Motion completion is a challenging and long-discussed problem, which is of great significance in film and game applications. For different motion completion scenarios (in-betweening, in-filling, and blending), most previous methods deal with the completion problems with case-by-case designs. In this work, we propose a simple but effective method to solve multiple motion completion problems under a unified framework and achieves a new state of the art accuracy under multiple evaluation settings. Inspired by the recent great success of attention-based models, we consider the completion as a sequence to sequence prediction problem. Our method consists of two modules - a standard transformer encoder with self-attention that learns long-range dependencies of input motions, and a trainable mixture embedding module that models temporal information and discriminates key-frames. Our method can run in a non-autoregressive manner and predict multiple missing frames within a single forward propagation in real time. We finally show the effectiveness of our method in music-dance applications.

Motivation & Objective

• Motivate and define motion completion across multiple scenarios (in-betweening, in-filling, blending) under a single framework.
• Propose a Transformer-based architecture with a trainable mixture embedding to model temporal information and keyframe roles.
• Enable non-autoregressive, one-shot prediction of multiple missing frames for real-time inference.
• Incorporate forward and inverse kinematics constraints to improve motion realism and consistency across coordinate systems.
• Evaluate on public datasets (LaFAN1, Anidance) and a new dance dataset, demonstrating state-of-the-art performance.

Proposed method

• Use a standard transformer encoder (BERT-style) as backbone to process a masked input sequence.
• Introduce a trainable mixture embedding combining a learnable position embedding and a keyframe embedding to annotate frames.
• Convert motion poses to sequential tokens via a Conv1d temporal operator before Transformer processing.
• Predict missing frames in a single forward pass, enabling non-autoregressive, parallel inference.
• Train with multi-task regression losses including pose reconstruction loss and kinematics losses (FK/IK) to enforce physical plausibility.
• Output a final predicted motion via a 1D convolution head after the Transformer stack.

Experimental results

Research questions

• RQ1 Can a unified Transformer-based framework handle in-betweening, in-filling, and blending in motion completion?
• RQ2 Does a trainable mixture embedding improve temporal modeling and keyframe discrimination for completion tasks?
• RQ3 Can non-autoregressive inference achieve real-time, multi-frame completion without sacrificing accuracy?
• RQ4 How do FK and IK losses impact accuracy when operating in global versus local coordinate systems?
• RQ5 How does the proposed approach perform on standard benchmarks (LaFAN1) and real-world/creative datasets (Anidance, dance blending)?

Key findings

• The method achieves state-of-the-art accuracy on LaFAN1 across multiple settings.
• Non-autoregressive single forward propagation enables real-time inference on CPU (e.g., 1x30 sequence in ~0.025s).
• Mixture embedding and IK/FK losses significantly improve accuracy across L2Q, L2P, and NPSS metrics.
• Global-coordinate predictions with the proposed losses generally yield higher accuracy than local-coordinate setups.
• The approach generalizes to in-betweening, in-filling, and blending, including in-wild keyframe arrangements.
• Qualitative results show coherent, plausible dance motion improvements over linear interpolation baselines.