[Paper Review] NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Human Pose and Shape Estimation

Jiefeng Li, Siyuan Bian+4 more|arXiv (Cornell University)|May 15, 2023

Human Pose and Action Recognition4 citations

TL;DR

NIKI introduces a novel invertible neural network-based inverse kinematics framework that jointly models forward and inverse kinematics to achieve both occlusion robustness and pixel-aligned 3D human pose and shape accuracy. By decoupling pose errors from plausible human postures via bijective error embedding and enforcing zero-error boundary conditions, NIKI outperforms state-of-the-art methods on both standard and occlusion-heavy benchmarks, achieving SOTA performance in accuracy and robustness simultaneously.

ABSTRACT

With the progress of 3D human pose and shape estimation, state-of-the-art methods can either be robust to occlusions or obtain pixel-aligned accuracy in non-occlusion cases. However, they cannot obtain robustness and mesh-image alignment at the same time. In this work, we present NIKI (Neural Inverse Kinematics with Invertible Neural Network), which models bi-directional errors to improve the robustness to occlusions and obtain pixel-aligned accuracy. NIKI can learn from both the forward and inverse processes with invertible networks. In the inverse process, the model separates the error from the plausible 3D pose manifold for a robust 3D human pose estimation. In the forward process, we enforce the zero-error boundary conditions to improve the sensitivity to reliable joint positions for better mesh-image alignment. Furthermore, NIKI emulates the analytical inverse kinematics algorithms with the twist-and-swing decomposition for better interpretability. Experiments on standard and occlusion-specific benchmarks demonstrate the effectiveness of NIKI, where we exhibit robust and well-aligned results simultaneously. Code is available at https://github.com/Jeff-sjtu/NIKI

Motivation & Objective

• Address the trade-off between mesh-image alignment and robustness to occlusions in 3D human pose and shape estimation.
• Overcome limitations of existing pixel-aligned methods, which fail under occlusions, and direct regression methods, which lack alignment precision.
• Develop a bi-directional learning framework that models both forward and inverse kinematics using invertible neural networks.
• Improve interpretability by emulating analytical inverse kinematics via twist-and-swing decomposition.
• Achieve state-of-the-art performance in both non-occluded and highly occluded scenarios.

Proposed method

• Employ invertible neural networks (INNs) to learn a bijective mapping between joint positions and the combined space of 3D joint rotations and latent error embeddings.
• In the inverse process, separate the error from the plausible human pose manifold by supervising the marginal distribution of the latent error, enhancing robustness to noisy or occluded joint detections.
• In the forward process, enforce zero-error boundary conditions to ensure that predicted rotations accurately explain reliable joint positions, improving mesh-image alignment.
• Decompose full-body rotations into twist and swing components using two consecutive invertible networks, improving interpretability and modeling fidelity.
• Train the INN jointly in both forward and inverse directions, leveraging the deterministic nature of forward kinematics to stabilize learning.
• Integrate heatmap conditioning during inference to further refine keypoint supervision, though performance is slightly reduced under severe occlusions.

Experimental results

Research questions

• RQ1 Can a unified framework simultaneously achieve high pixel-aligned accuracy and robustness to occlusions in 3D human pose and shape estimation?
• RQ2 How can invertible neural networks be leveraged to model bi-directional error propagation in inverse kinematics for improved robustness?
• RQ3 To what extent does enforcing zero-error boundary conditions in the forward process enhance mesh-image alignment without sacrificing robustness?
• RQ4 Does twist-and-swing decomposition improve interpretability and performance in neural inverse kinematics compared to end-to-end rotation prediction?
• RQ5 How does the proposed method perform across varying levels of occlusion, especially in extreme cases?

Key findings

• NIKI achieves a 71.3 mm MPJPE and 40.6 mm PA-MPJPE on the standard 3DPW benchmark, outperforming prior state-of-the-art methods.
• On the challenging 3DPW-XOCC dataset with extreme occlusions, NIKI achieves 110.7 mm MPJPE and 60.5 mm PA-MPJPE, demonstrating strong robustness.
• NIKI reduces the mesh collision ratio from 2.6% to 1.0% on 3DPW-XOCC, indicating more plausible and realistic reconstructions.
• In medium occlusion levels (20–30%), NIKI achieves a 13.3 mm MPJPE improvement over PARE, highlighting its effectiveness in realistic occlusion scenarios.
• The method shows a 6.5 mm MPJPE improvement in low occlusion levels (0–10%) and maintains a 10.2 mm improvement in high occlusion levels (70–80%), confirming consistent performance across occlusion intensities.
• Despite adding heatmap conditioning, NIKI's performance slightly degrades on 3DPW-XOCC, suggesting that heatmap supervision is less reliable under severe occlusions.