[Paper Review] SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach

Ailing Zeng, Xiao Sun+4 more|arXiv (Cornell University)|Jul 18, 2020

Human Pose and Action RecognitionComputer Science41 references17 citations

TL;DR

SRNet improves 3D human pose estimation generalization by splitting the body into local joint groups processed in separate branches, while recombining low-dimensional global context to maintain coherence. This approach reduces the impact of long-tailed pose distributions, significantly boosting performance on rare and unseen poses, especially in video-based settings with state-of-the-art results on Human3.6M and 3DHP datasets.

ABSTRACT

Human poses that are rare or unseen in a training set are challenging for a network to predict. Similar to the long-tailed distribution problem in visual recognition, the small number of examples for such poses limits the ability of networks to model them. Interestingly, local pose distributions suffer less from the long-tail problem, i.e., local joint configurations within a rare pose may appear within other poses in the training set, making them less rare. We propose to take advantage of this fact for better generalization to rare and unseen poses. To be specific, our method splits the body into local regions and processes them in separate network branches, utilizing the property that a joint position depends mainly on the joints within its local body region. Global coherence is maintained by recombining the global context from the rest of the body into each branch as a low-dimensional vector. With the reduced dimensionality of less relevant body areas, the training set distribution within network branches more closely reflects the statistics of local poses instead of global body poses, without sacrificing information important for joint inference. The proposed split-and-recombine approach, called SRNet, can be easily adapted to both single-image and temporal models, and it leads to appreciable improvements in the prediction of rare and unseen poses.

Motivation & Objective

• To address the challenge of poor generalization in 3D human pose estimation for rare or unseen global poses.
• To mitigate the long-tailed distribution problem in training data by leveraging locally frequent joint configurations.
• To improve model robustness to unseen poses without relying on synthetic data augmentation.
• To maintain global pose coherence while enhancing local feature learning through structured network decomposition.
• To achieve state-of-the-art performance on both single-frame and temporal 3D pose estimation benchmarks.

Proposed method

• The network splits the human body into local joint groups with strong internal dependencies, each processed in a dedicated branch.
• Each branch learns local joint configurations independently, reducing the influence of less relevant global body regions.
• A low-dimensional global context vector is computed from other branches and recombined into each branch to preserve pose coherence.
• Dimensionality reduction of the global context minimizes interference from irrelevant body parts during local feature learning.
• The architecture is compatible with both single-image and temporal models, enabling application to video-based pose estimation.
• The method does not require additional data augmentation and leverages only the original training data more effectively.

Experimental results

Research questions

• RQ1 Can local joint configurations that are frequent in the training set be leveraged to improve generalization to rare or unseen global poses?
• RQ2 Does splitting the body into local regions and recombining global context improve performance on rare poses compared to standard end-to-end networks?
• RQ3 Can this split-and-recombine approach outperform existing state-of-the-art methods on both single-frame and video-based 3D pose estimation?
• RQ4 How does the dimensionality of the global context vector affect performance and generalization?
• RQ5 To what extent does the method reduce the impact of long-tailed pose distributions in training data?

Key findings

• SRNet achieves a 12.2% relative improvement (11.4mm reduction) on the Cross Action protocol of Human3.6M compared to the prior state-of-the-art method [7].
• On the PI-3DHP test set, SRNet achieves a mean MPJPE of 82.0mm, outperforming all prior methods in the comparison.
• With 2D ground truth input, SRNet achieves 32.0mm MPJPE on Human3.6M, a 14.0% relative improvement over [27] (37.2mm).
• SRNet reduces MPJPE by 4.3% (from 46.8mm to 44.8mm) when using 2D keypoint detection as input, demonstrating strong robustness to detection errors.
• The model uses only 3.61M parameters, about one-fifth of the 16.95M parameters in [27], indicating high efficiency.
• Ablation studies confirm that the split-and-recombine design significantly improves performance on rare actions, especially those with low training frequency.