[Paper Review] IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions

Anindita Ghosh, Rishabh Dabral+3 more|arXiv (Cornell University)|Dec 14, 2022

Human Pose and Action Recognition4 citations

TL;DR

IMoS proposes a novel intent-driven framework for synthesizing diverse, full-body motion sequences of virtual characters interacting with 3D objects based on natural language instructions. It uses decoupled conditional variational autoregressors for arms and body, jointly optimizing 6-DoF object poses to enable plausible, diverse human-object interactions, achieving state-of-the-art results in intent-driven motion synthesis.

ABSTRACT

Can we make virtual characters in a scene interact with their surrounding objects through simple instructions? Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions? Inspired by these questions, we present the first framework to synthesize the full-body motion of virtual human characters performing specified actions with 3D objects placed within their reach. Our system takes textual instructions specifying the objects and the associated intentions of the virtual characters as input and outputs diverse sequences of full-body motions. This contrasts existing works, where full-body action synthesis methods generally do not consider object interactions, and human-object interaction methods focus mainly on synthesizing hand or finger movements for grasping objects. We accomplish our objective by designing an intent-driven fullbody motion generator, which uses a pair of decoupled conditional variational auto-regressors to learn the motion of the body parts in an autoregressive manner. We also optimize the 6-DoF pose of the objects such that they plausibly fit within the hands of the synthesized characters. We compare our proposed method with the existing methods of motion synthesis and establish a new and stronger state-of-the-art for the task of intent-driven motion synthesis.

Motivation & Objective

• To enable the synthesis of diverse, full-body motion sequences for virtual characters performing specified actions with 3D objects based on natural language instructions.
• To address the gap in existing methods that focus only on hand or finger movements, or fail to model full-body coordination in human-object interactions.
• To generate plausible, physically consistent motions that include coordinated head, torso, arm, and hand movements without explicit supervision on head orientation.
• To optimize 6-DoF object poses in real time to ensure accurate and stable grasping during motion sequences.
• To support both single-handed and two-handed interactions, including off-handing transitions, within a unified generative framework.

Proposed method

• The method employs a pair of decoupled conditional variational autoregressors—one for arms and one for the rest of the body—to model motion generation in separate latent spaces for improved precision.
• A Condition Encoder, based on CLIP, encodes textual instructions (intent and object type) to condition the autoregressive motion generation process.
• The model uses an autoregressive, variational inference-based approach to generate future body poses from a short history, enabling diverse motion sampling from the latent space.
• An Object Optimizer Module jointly optimizes the 6-DoF pose of the object to satisfy grasping constraints, ensuring physical plausibility and contact fidelity.
• The framework supports both static grasps and dynamic transitions such as off-handing, where the object is passed between hands, by detecting switching frames and transferring hand parameters.
• The model is trained on the GRAB dataset and leverages sparse marker-based motion capture, with post-processing to refine object-hand contact and reduce inter-penetration.

Experimental results

Research questions

• RQ1 Can a single framework generate diverse, full-body motion sequences for human-object interactions from natural language instructions alone?
• RQ2 Does decoupling arm and body motion modeling improve the quality and precision of synthesized interactions compared to joint modeling?
• RQ3 Can the model generate plausible head and torso coordination without explicit supervision on head direction?
• RQ4 How effective is the 6-DoF object pose optimization in achieving stable and contact-consistent grasps during motion sequences?
• RQ5 Can the framework generalize to complex interactions such as off-handing and dynamic grasps, including those with high-frequency wrist motions?

Key findings

• The proposed method achieves a new state-of-the-art in intent-driven full-body motion synthesis, outperforming existing methods on both quantitative metrics and perceptual evaluation.
• The decoupled autoregressive modeling of arms and body leads to measurable perceptual improvements, especially in fine-grained hand and arm coordination.
• The model generates diverse motion sequences from the same instruction by sampling from the variational latent space, demonstrating strong generalization and expressiveness.
• Despite no explicit supervision, the model learns visually plausible head orientations and coordinated hand-head motion, suggesting that such coordination can emerge from end-to-end training.
• The Object Optimizer Module successfully reduces inter-penetration and improves contact accuracy, though challenges remain for dynamic grasps like rotating or stretching objects.
• Failure cases occur for rare, high-frequency actions such as "squeeze" or "shake", indicating limitations in capturing rapid wrist motions.