[Paper Review] Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation

Jogendra Nath Kundu, Siddharth Seth+5 more|arXiv (Cornell University)|Apr 4, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a self-supervised 3D human pose estimation framework that distills non-local latent relations between unpaired image and 3D pose sequences to enable domain adaptation without paired supervision. By leveraging a pre-trained generative adversarial auto-encoder for pose plausibility and introducing higher-order, non-local relations in pose and motion spaces, the method achieves state-of-the-art performance on standard benchmarks like 3DPW and 3DHP, reducing MPJPE to 86.2 mm under unseen transfer settings.

ABSTRACT

Available 3D human pose estimation approaches leverage different forms of strong (2D/3D pose) or weak (multi-view or depth) paired supervision. Barring synthetic or in-studio domains, acquiring such supervision for each new target environment is highly inconvenient. To this end, we cast 3D pose learning as a self-supervised adaptation problem that aims to transfer the task knowledge from a labeled source domain to a completely unpaired target. We propose to infer image-to-pose via two explicit mappings viz. image-to-latent and latent-to-pose where the latter is a pre-learned decoder obtained from a prior-enforcing generative adversarial auto-encoder. Next, we introduce relation distillation as a means to align the unpaired cross-modal samples i.e. the unpaired target videos and unpaired 3D pose sequences. To this end, we propose a new set of non-local relations in order to characterize long-range latent pose interactions unlike general contrastive relations where positive couplings are limited to a local neighborhood structure. Further, we provide an objective way to quantify non-localness in order to select the most effective relation set. We evaluate different self-adaptation settings and demonstrate state-of-the-art 3D human pose estimation performance on standard benchmarks.

Motivation & Objective

• To address the challenge of domain shift in 3D human pose estimation by enabling self-supervised adaptation from labeled source domains to unlabeled target domains.
• To overcome the limitations of paired supervision, which is costly and impractical in real-world settings like in-the-wild environments.
• To develop a method that aligns unpaired image and 3D pose data through structured, instance-grounded relational knowledge in latent space.
• To reduce instance-level misalignment in representation learning by modeling long-range, non-local interactions in pose and motion sequences.
• To achieve state-of-the-art performance on standard benchmarks without relying on multi-view, depth, or 2D keypoint supervision.

Proposed method

• The framework uses two explicit mappings: image-to-latent CNN and a pre-learned latent-to-pose decoder from a prior-enforcing GAN-based auto-encoder (AAE).
• A recurrent AAE is used to pre-train motion embeddings on unpaired 3D pose sequences, ensuring temporal consistency and plausibility.
• Non-local relations—such as 'pose-flip' and 'flip-backward'—are introduced to model long-range interactions beyond local neighborhoods in pose and motion spaces.
• Relational energy functions are learned via frozen neural networks that regress the latent embedding of relational counterparts (e.g., flipped or reversed sequences).
• The image-to-latent network is adapted via distillation of these relational energies, minimizing the discrepancy between image and pose latent manifolds.
• A progressive training schedule applies multiple relational losses: $\mathcal{L}_{LCR}$, $\mathcal{L}_{HCR}$, $\mathcal{L}_{3}^{z}$, $\mathcal{L}_{2}^{v}$, and $\mathcal{L}_{3}^{v}$, to progressively refine alignment.

Experimental results

Research questions

• RQ1 Can non-local, higher-order relations in latent pose and motion spaces effectively align unpaired image and 3D pose data without paired supervision?
• RQ2 How does modeling long-range interactions improve representation alignment and reduce instance-level misalignment in self-supervised 3D pose estimation?
• RQ3 To what extent does the proposed framework generalize across diverse domains, including in-the-wild and in-studio settings?
• RQ4 Can a pre-trained generative decoder enforce plausibility while enabling effective adaptation to unlabeled target domains?
• RQ5 How do different relational losses contribute to the final performance, and which combination yields optimal generalization?

Key findings

• The proposed method, Ours(SH → W), achieves a state-of-the-art MPJPE of 86.2 mm on the 3DPW benchmark under unseen transfer, outperforming prior methods.
• Adding both non-local pose and motion relation distillation ($\mathcal{L}_{3}^{z}$ and $\mathcal{L}_{3}^{v}$) reduces MPJPE from 209.6 mm (source-only) to 86.2 mm, demonstrating significant improvement.
• The ablation study confirms that distilling higher-order non-local motion relations ($\mathcal{L}_{2}^{v}$ and $\mathcal{L}_{3}^{v}$) is critical for reducing instance-level misalignment.
• The use of in-studio Human3.6M data as an additional labeled source (SH → W) significantly reduces domain gap and improves generalization to in-the-wild data.
• Qualitative results show robust generalization across diverse poses, clothing, and backgrounds, with failures primarily occurring in cases of severe occlusion or rare athletic poses.
• The framework maintains plausible 3D pose outputs due to the prior-enforcing generative decoder, even in the absence of paired supervision.