[Paper Review] 3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning

Xiangyu Xu, Hao Chen+3 more|arXiv (Cornell University)|Jul 27, 2020

Advanced Vision and Imaging54 references4 citations

TL;DR

This paper proposes RSC-Net, a resolution-aware deep neural network that estimates 3D human shape and pose from single low-resolution images using self-supervised and contrastive learning. By enforcing scale-consistent outputs and features across resolutions, RSC-Net achieves state-of-the-art performance without requiring high-resolution supervision or super-resolution preprocessing.

ABSTRACT

3D human shape and pose estimation from monocular images has been an active area of research in computer vision, having a substantial impact on the development of new applications, from activity recognition to creating virtual avatars. Existing deep learning methods for 3D human shape and pose estimation rely on relatively high-resolution input images; however, high-resolution visual content is not always available in several practical scenarios such as video surveillance and sports broadcasting. Low-resolution images in real scenarios can vary in a wide range of sizes, and a model trained in one resolution does not typically degrade gracefully across resolutions. Two common approaches to solve the problem of low-resolution input are applying super-resolution techniques to the input images which may result in visual artifacts, or simply training one model for each resolution, which is impractical in many realistic applications. To address the above issues, this paper proposes a novel algorithm called RSC-Net, which consists of a Resolution-aware network, a Self-supervision loss, and a Contrastive learning scheme. The proposed network is able to learn the 3D body shape and pose across different resolutions with a single model. The self-supervision loss encourages scale-consistency of the output, and the contrastive learning scheme enforces scale-consistency of the deep features. We show that both these new training losses provide robustness when learning 3D shape and pose in a weakly-supervised manner. Extensive experiments demonstrate that the RSC-Net can achieve consistently better results than the state-of-the-art methods for challenging low-resolution images.

Motivation & Objective

• Address the challenge of 3D human shape and pose estimation from low-resolution images, which are common in real-world applications like surveillance and sports broadcasting.
• Overcome the limitations of existing methods that rely on high-resolution inputs or suffer from poor generalization across resolutions.
• Mitigate the lack of high-quality 3D annotations in real-world datasets by leveraging self-supervised learning with resolution-consistent supervision.
• Improve feature representation robustness across resolutions using contrastive learning to maximize mutual information between features at different scales.
• Develop a single-model solution that generalizes across arbitrary input resolutions without requiring separate models or super-resolution pre-processing.

Proposed method

• Propose a resolution-aware network architecture that shares a feature extractor across resolutions while using resolution-specific parameters to adaptively integrate features.
• Introduce a directional self-supervision loss that enforces consistency between predictions of the same image at different resolutions, using higher-resolution outputs as soft supervision for lower-resolution inputs.
• Design a contrastive feature loss based on cosine similarity to maximize mutual information between deep features extracted from the same image at different resolutions, improving feature robustness.
• Use progressive training to stabilize optimization when learning from multi-resolution inputs, gradually introducing lower-resolution data.
• Adopt hierarchical supervision in the self-supervision loss to guide low-resolution predictions using predictions from higher resolutions, improving training stability and accuracy.
• Combine the resolution-aware network, self-supervision loss, and contrastive feature loss into a unified training objective for weakly-supervised 3D estimation.

Experimental results

Research questions

• RQ1 Can a single deep neural network effectively estimate 3D human shape and pose across a wide range of low-resolution inputs without retraining or super-resolution?
• RQ2 How can self-supervision be effectively leveraged to improve 3D estimation when high-quality 3D annotations are scarce or unavailable?
• RQ3 Does enforcing feature consistency across resolutions via contrastive learning lead to better generalization and robustness compared to standard MSE-based losses?
• RQ4 Can a directional self-supervision strategy that prioritizes high-resolution guidance improve performance on low-resolution inputs compared to symmetric supervision?
• RQ5 How does progressive training impact the convergence and performance of a multi-resolution 3D estimation model?

Key findings

• RSC-Net achieves state-of-the-art performance on low-resolution 3D human pose and shape estimation, outperforming existing methods across all tested resolutions.
• The combination of resolution-aware network, self-supervision loss, and contrastive feature loss reduces MPJPE to 96.36 on the lowest resolution (32×32), significantly outperforming the baseline (105.11) without progressive training.
• The directional self-supervision loss (8) improves results by 8.75 MPJPE on 32×32 resolution compared to the original symmetric self-supervision loss (143.31 vs. 105.11), demonstrating its effectiveness in leveraging high-resolution guidance.
• Contrastive feature loss (CL) outperforms MSE-based feature consistency loss (MS), reducing MPJPE by 11.44 on 32×32 resolution (117.12 vs. 128.56), confirming its superiority in high-dimensional feature alignment.
• Progressive training is critical: skipping it increases MPJPE by 8.75 on 32×32 resolution (127.05 vs. 117.12), showing its role in stabilizing training on multi-resolution data.
• The full model (RA+SS+CL) achieves 58.98 MPJPE-PA on 32×32 resolution, demonstrating robustness and generalization across extreme low-resolution inputs without super-resolution preprocessing.