[Paper Review] InsActor: Instruction-driven Physics-based Characters

Jiawei Ren, Mingyuan Zhang+4 more|arXiv (Cornell University)|Dec 28, 2023

Human Motion and Animation4 citations

TL;DR

InsActor is a hierarchical diffusion-based framework that generates physically plausible, instruction-driven animations for physics-based characters by combining language-conditioned motion planning with learned low-level skills. It achieves state-of-the-art performance in instruction-following and waypoint navigation, with a 90.7% success rate on complex tasks, by mapping high-level instructions to latent skill sequences that ensure physical feasibility and motion coherence.

ABSTRACT

Generating animation of physics-based characters with intuitive control has long been a desirable task with numerous applications. However, generating physically simulated animations that reflect high-level human instructions remains a difficult problem due to the complexity of physical environments and the richness of human language. In this paper, we present InsActor, a principled generative framework that leverages recent advancements in diffusion-based human motion models to produce instruction-driven animations of physics-based characters. Our framework empowers InsActor to capture complex relationships between high-level human instructions and character motions by employing diffusion policies for flexibly conditioned motion planning. To overcome invalid states and infeasible state transitions in planned motions, InsActor discovers low-level skills and maps plans to latent skill sequences in a compact latent space. Extensive experiments demonstrate that InsActor achieves state-of-the-art results on various tasks, including instruction-driven motion generation and instruction-driven waypoint heading. Notably, the ability of InsActor to generate physically simulated animations using high-level human instructions makes it a valuable tool, particularly in executing long-horizon tasks with a rich set of instructions.

Motivation & Objective

• To enable physically simulated character animation controlled by high-level human instructions, which are complex and diverse in natural language.
• To address the challenge of generating motion plans that are both semantically aligned with instructions and physically feasible in dynamic environments.
• To overcome limitations of direct diffusion policy outputs, which may produce infeasible or unstable state transitions in physics simulations.
• To develop a scalable, flexible framework that supports long-horizon tasks and interactive conditions like waypoints without task-specific retraining.
• To establish a new baseline for language-conditioned, physics-based character animation with improved robustness and generalization.

Proposed method

• Uses a diffusion policy to generate high-level motion plans conditioned on natural language instructions, enabling flexible test-time conditioning.
• Employs an unsupervised skill discovery module with an encoder-decoder architecture to map joint-space trajectories into a compact latent skill space.
• Encodes infeasible or unstable state transitions from the high-level policy into latent skill vectors to ensure physical plausibility.
• Decodes the learned skill embeddings back into executable actions using an inverse dynamics-compatible policy.
• Supports autoregressive conditioning on history motion for interactive, multi-waypoint navigation tasks.
• Introduces a learnable weight factor λ to balance skill space compactness and instruction-motion alignment.

Experimental results

Research questions

• RQ1 Can a diffusion-based policy generate high-level motion plans from natural language instructions that are both semantically aligned and physically feasible?
• RQ2 How can infeasible state transitions from a high-level policy be corrected to ensure stable, physically plausible execution in simulation?
• RQ3 To what extent does hierarchical decomposition into high-level planning and low-level skill execution improve motion quality and task success in instruction-driven animation?
• RQ4 Can the framework generalize to long-horizon tasks and interactive conditions like multiple waypoints without fine-tuning?
• RQ5 How does the trade-off between skill space compactness and instruction alignment affect overall performance?

Key findings

• InsActor achieves a 90.7% success rate on complex instruction-following tasks, significantly outperforming baselines such as Diffuser (38.0%) and other methods.
• The hierarchical design with skill discovery improves motion fidelity and reduces error accumulation, achieving high R-precision and low FID scores.
• Ablation studies confirm that combining high-level planning with low-level skill execution is essential—using only the high-level policy leads to poor instruction alignment, while only low-level policies fail to understand language context.
• The weight factor λ in skill space regularization enables a tunable trade-off between motion diversity and physical plausibility, with optimal performance at balanced settings.
• Qualitative results show that InsActor generates visually plausible, diverse, and coherent animations, including multi-waypoint navigation and complex motion sequences.
• The framework supports flexible conditioning with additional inputs like waypoints, enabling interactive control without retraining.