[Paper Review] HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery

Jiefeng Li, Siyuan Bian+4 more|arXiv (Cornell University)|Apr 12, 2023

3D Shape Modeling and AnalysisEngineering14 citations

TL;DR

HybrIK-X introduces a hybrid analytical-neural inverse kinematics framework to recover pixel-aligned whole-body meshes by transforming accurate 3D joints into body-part rotations, extended to hands and face with a one-stage model.

ABSTRACT

Recovering whole-body mesh by inferring the abstract pose and shape parameters from visual content can obtain 3D bodies with realistic structures. However, the inferring process is highly non-linear and suffers from image-mesh misalignment, resulting in inaccurate reconstruction. In contrast, 3D keypoint estimation methods utilize the volumetric representation to achieve pixel-level accuracy but may predict unrealistic body structures. To address these issues, this paper presents a novel hybrid inverse kinematics solution, HybrIK, that integrates the merits of 3D keypoint estimation and body mesh recovery in a unified framework. HybrIK directly transforms accurate 3D joints to body-part rotations via twist-and-swing decomposition. The swing rotations are analytically solved with 3D joints, while the twist rotations are derived from visual cues through neural networks. To capture comprehensive whole-body details, we further develop a holistic framework, HybrIK-X, which enhances HybrIK with articulated hands and an expressive face. HybrIK-X is fast and accurate by solving the whole-body pose with a one-stage model. Experiments demonstrate that HybrIK and HybrIK-X preserve both the accuracy of 3D joints and the realistic structure of the parametric human model, leading to pixel-aligned whole-body mesh recovery. The proposed method significantly surpasses the state-of-the-art methods on various benchmarks for body-only, hand-only, and whole-body scenarios. Code and results can be found at https://jeffli.site/HybrIK-X/

Motivation & Objective

• Address the gap between accurate 3D joints and realistic body meshes by bridging 3D keypoint estimation with parametric body models.
• Develop a hybrid IK solver that uses analytic swing rotations and neural twist angles to recover body-part rotations.
• Extend the approach to whole-body mesh recovery including hands and face in a single-stage framework.
• Improve robustness to occlusions/truncations and reduce computational complexity compared to multi-expert pipelines.

Proposed method

• Use twist-and-swing decomposition to factor body-part rotations into swing (analytically solved) and twist (neural) components.
• Solve rotations along the kinematic tree in a differentiable, feed-forward manner.
• Extend to HybrIK-X by dividing the body into sub-trees (body, hands, face) and applying backward-updated merge to ensure consistency at conflict joints.
• Leverage SMPL/SMPL-X models for rest pose, shape beta, and expression psi to produce a differentiable mesh via linear blend skinning.
• Train end-to-end by regressing 2.5D joints, twist angles, shape, and expression parameters, then recover full mesh through IK and FK.

Experimental results

Research questions

• RQ1 Can a hybrid analytical-neural IK pipeline yield pixel-aligned whole-body mesh recovery from monocular images?
• RQ2 Does twist-and-swing IK enable more stable and realistic body-part rotations than direct regression of full 3D rotations?
• RQ3 Can a one-stage model reliably recover body, hands, and face without separate expert networks while handling occlusions and truncations?
• RQ4 How does backward-updated merging across sub-trees affect consistency at conflict joints in whole-body reconstruction?

Key findings

• HybrIK-X achieves pixel-aligned whole-body mesh recovery by solving body-part rotations with twist-and-swing IK.
• The swing component is solved analytically from 3D joints, while the twist component is predicted by a neural network, enabling differentiable end-to-end training.
• HybrIK-X extends HybrIK to hands and face in a one-stage framework, improving efficiency over multi-expert pipelines.
• Backward-updated merging across sub-trees reduces conflicts at shared joints, enabling coherent whole-body reconstructions.
• The method significantly surpasses state-of-the-art approaches on body-only, hand-only, and whole-body benchmarks (as claimed).