[Paper Review] GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency

Omid Taheri, Yi Zhou+5 more|UvA-DARE (University of Amsterdam)|Aug 22, 2023

Human Pose and Action RecognitionComputer Science3 citations

TL;DR

GRIP is a learning-based method that generates realistic, temporally consistent hand-object interaction poses from noisy 3D body and object motion. It uses denoised arm motion, novel distance-based hand sensors (Ambient and Proximity), and a two-stage inference pipeline with latent-space temporal consistency to synthesize dynamic, penetration-free hand poses for single- and bi-manual grasps, outperforming baselines in generalization and realism across unseen objects and datasets.

ABSTRACT

Hands are dexterous and highly versatile manipulators that are central to how humans interact with objects and their environment. Consequently, modeling realistic hand-object interactions, including the subtle motion of individual fingers, is critical for applications in computer graphics, computer vision, and mixed reality. Prior work on capturing and modeling humans interacting with objects in 3D focuses on the body and object motion, often ignoring hand pose. In contrast, we introduce GRIP, a learning-based method that takes, as input, the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction. As a preliminary step before synthesizing the hand motion, we first use a network, ANet, to denoise the arm motion. Then, we leverage the spatio-temporal relationship between the body and the object to extract two types of novel temporal interaction cues, and use them in a two-stage inference pipeline to generate the hand motion. In the first stage, we introduce a new approach to enforce motion temporal consistency in the latent space (LTC), and generate consistent interaction motions. In the second stage, GRIP generates refined hand poses to avoid hand-object penetrations. Given sequences of noisy body and object motion, GRIP upgrades them to include hand-object interaction. Quantitative experiments and perceptual studies demonstrate that GRIP outperforms baseline methods and generalizes to unseen objects and motions from different motion-capture datasets.

Motivation & Objective

• To address the lack of realistic, dynamic hand motion in 3D human-object interaction animations, especially when hand data is missing or noisy.
• To enable automatic synthesis of temporally consistent, physically plausible hand poses during pre-grasp, grasp, and post-grasp phases.
• To generalize across diverse object shapes, sizes, and motion types without requiring hand-specific annotations.
• To achieve real-time inference by avoiding iterative optimization, enabling practical deployment in animation and XR pipelines.
• To upgrade existing motion-capture datasets by adding accurate, naturalistic hand interactions without re-capturing hand data.

Proposed method

• A denoising network, ANet, is used to refine noisy arm motion from input body and object sequences.
• Two novel virtual hand sensors are introduced: the Ambient Sensor captures object shape and motion within the hand's broader reach, and the Proximity Sensor encodes fine-grained geometric and distance features between hand and object surfaces.
• A two-stage inference pipeline is employed: first, the Consistency Network (CNet) generates temporally consistent hand poses in latent space using spatio-temporal cues.
• Second, the Refinement Network (RNet) improves pose accuracy by re-computing proximity features and reducing hand-object penetrations.
• Latent-space temporal consistency (LTC) is enforced in CNet to ensure smooth, natural motion transitions across frames.
• The method is trained and evaluated on multiple datasets (GRAB, InterCap), with generalization tested on unseen objects and motion types.

Experimental results

Research questions

• RQ1 Can a learning-based method generate realistic, dynamic hand-object interaction poses from only body and object motion, without requiring hand annotations?
• RQ2 How can spatio-temporal interaction cues between hands, body, and objects be effectively modeled to guide hand pose generation?
• RQ3 Can latent-space temporal consistency improve the realism and smoothness of generated hand motion compared to post-hoc smoothing?
• RQ4 To what extent can the model generalize to unseen object shapes and novel motion sequences from different datasets?
• RQ5 Can the method reduce hand-object penetrations while maintaining high temporal consistency and real-time performance?

Key findings

• GRIP reduces penetration volume to 2.38 cm³ (vs. 2.65 cm³ for GrabNet and 2.68 cm³ for ManipNet), demonstrating superior accuracy in avoiding hand-object intersections.
• The contact ratio is 1.00 across all test sequences, indicating that GRIP maintains consistent and plausible contact between hands and objects.
• GRIP achieves a MPVPE (Mean Per-Vertex Position Error) of 6.17 mm for the left hand and 7.88 mm for the right hand, outperforming baselines like GrabNet (8.18 mm) and ManipNet (7.78 mm).
• The full GRIP model (with RNet) achieves a 10% improvement in penetration ratio and better generalization to unseen objects and motions compared to state-of-the-art baselines.
• The two-stage pipeline with LTC and refinement significantly improves motion quality, with ablation studies showing that RNet reduces penetration by 25% compared to GRIP without RNet.
• GRIP generalizes effectively to unseen datasets (e.g., InterCap) and object types, including complex bi-manual grasps and delicate pinching motions.