[Paper Review] Inference Stage Optimization for Cross-scenario 3D Human Pose Estimation

Jianfeng Zhang, Xuecheng Nie+1 more|arXiv (Cornell University)|Jul 4, 2020

Human Pose and Action RecognitionComputer Science55 references19 citations

TL;DR

This paper proposes Inference Stage Optimization (ISO), a novel framework that enhances 3D human pose estimation generalization in cross-scenario settings by performing geometry-aware self-supervised learning (SSL) on unlabeled target data during inference. By leveraging random projection adversaries and geometric cycle consistency, ISO adapts the model instance-wisely before prediction, achieving state-of-the-art 83.6% 3D PCK on MPI-INF-3DHP, a 9.7% improvement over prior methods.

ABSTRACT

Existing 3D human pose estimation models suffer performance drop when applying to new scenarios with unseen poses due to their limited generalizability. In this work, we propose a novel framework, Inference Stage Optimization (ISO), for improving the generalizability of 3D pose models when source and target data come from different pose distributions. Our main insight is that the target data, even though not labeled, carry valuable priors about their underlying distribution. To exploit such information, the proposed ISO performs geometry-aware self-supervised learning (SSL) on each single target instance and updates the 3D pose model before making prediction. In this way, the model can mine distributional knowledge about the target scenario and quickly adapt to it with enhanced generalization performance. In addition, to handle sequential target data, we propose an online mode for implementing our ISO framework via streaming the SSL, which substantially enhances its effectiveness. We systematically analyze why and how our ISO framework works on diverse benchmarks under cross-scenario setup. Remarkably, it yields new state-of-the-art of 83.6% 3D PCK on MPI-INF-3DHP, improving upon the previous best result by 9.7%. Code will be released.

Motivation & Objective

• To address the performance drop of 3D pose estimation models when generalizing to new scenarios with unseen pose distributions.
• To improve model generalization without requiring labeled data in the target domain.
• To develop a method that adapts the model during inference using priors from unlabeled target instances.
• To enable effective adaptation for sequential data through an online variant of the framework.
• To provide empirical and ablation-based understanding of why and how inference-stage adaptation improves cross-scenario performance.

Proposed method

• The ISO framework performs self-supervised learning (SSL) on each unlabeled target sample during inference to refine the model's predictions.
• It employs two geometry-aware SSL techniques: random projection adversary and geometric cycle consistency, to learn robust 3D pose representations.
• The model is jointly trained with fully-supervised learning (FSL) and SSL to improve generalization and enable effective inference-time adaptation.
• An online ISO variant streams SSL updates across sequential target samples, enabling real-time adaptation with reduced computational cost.
• The framework performs iterative optimization on each target instance using back-propagation to refine 3D pose estimates based on geometric priors.
• A lightweight inference mode (Online-skip) processes SSL only once every 10 samples, achieving near-real-time efficiency with minimal performance drop.

Experimental results

Research questions

• RQ1 Can self-supervised learning on unlabeled target data improve 3D pose estimation generalization in cross-scenario settings?
• RQ2 How do geometry-aware SSL components like random projection adversary and geometric cycle consistency enhance inference-time adaptation?
• RQ3 What is the impact of online adaptation on sequential data, and how does it compare to batch inference?
• RQ4 How does ISO perform under noisy 2D pose inputs, and does it outperform baseline models?
• RQ5 Why does ISO achieve better performance on specific body parts like the head, wrists, and ankles in challenging scenarios?

Key findings

• ISO achieves a new state-of-the-art 83.6% 3D PCK on the MPI-INF-3DHP benchmark, a 9.7% improvement over the previous best result.
• The online ISO variant achieves 83.6% 3D PCK with only 0.027 seconds per sample, enabling efficient real-time inference.
• The Online-skip variant reduces inference time to 0.004 seconds per sample—nearly matching standard inference—while maintaining 83.0% 3D PCK.
• ISO improves performance significantly on hard-to-estimate body parts such as the head, wrists, and ankles, indicating robustness to distribution shifts.
• Even under extreme Gaussian noise (σ=10), ISO maintains 79.6% 3D PCK, outperforming the baseline (78.9%) with ground-truth 2D poses.
• Distribution alignment analysis confirms that ISO successfully aligns the predicted 3D pose distribution with the ground-truth distribution on the target set.