[Paper Review] 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos

Xiangyu Xu, Hao Chen+3 more|arXiv (Cornell University)|Mar 11, 2021

Advanced Vision and Imaging74 references4 citations

TL;DR

This paper proposes RSC-Net, a resolution-aware deep neural network that estimates 3D human pose, shape, and texture from low-resolution images and videos using self-supervision and contrastive learning. It achieves state-of-the-art performance across multiple resolutions with a single model, overcoming limitations of super-resolution and resolution-specific training.

ABSTRACT

3D human pose and shape estimation from monocular images has been an active research area in computer vision. Existing deep learning methods for this task rely on high-resolution input, which however, is not always available in many scenarios such as video surveillance and sports broadcasting. Two common approaches to deal with low-resolution images are applying super-resolution techniques to the input, which may result in unpleasant artifacts, or simply training one model for each resolution, which is impractical in many realistic applications. To address the above issues, this paper proposes a novel algorithm called RSC-Net, which consists of a Resolution-aware network, a Self-supervision loss, and a Contrastive learning scheme. The proposed method is able to learn 3D body pose and shape across different resolutions with one single model. The self-supervision loss enforces scale-consistency of the output, and the contrastive learning scheme enforces scale-consistency of the deep features. We show that both these new losses provide robustness when learning in a weakly-supervised manner. Moreover, we extend the RSC-Net to handle low-resolution videos and apply it to reconstruct textured 3D pedestrians from low-resolution input. Extensive experiments demonstrate that the RSC-Net can achieve consistently better results than the state-of-the-art methods for challenging low-resolution images.

Motivation & Objective

• Address the challenge of 3D human pose and shape estimation from low-resolution images, which are common in surveillance and sports videos.
• Overcome the limitations of existing methods that rely on high-resolution inputs or super-resolution, which introduce artifacts and domain gaps.
• Develop a unified deep learning model that generalizes across arbitrary image resolutions without retraining or resolution-specific adaptation.
• Mitigate the issue of weak supervision due to sparse 3D annotations by introducing self-supervised and contrastive learning strategies.
• Extend the method to reconstruct textured 3D pedestrians from low-resolution video inputs.

Proposed method

• Proposes RSC-Net, a resolution-aware network with a shared feature extractor and resolution-dependent parameters to adaptively fuse features across different resolutions.
• Introduces a directional self-supervision loss that enforces scale-consistency between predictions from the same image at different resolutions, using higher-resolution outputs as guidance.
• Employs a contrastive learning scheme to enforce consistency in deep feature representations across resolutions, using a contrastive loss that maximizes mutual information between features.
• Uses a hierarchical supervision strategy in the self-supervision loss, where high-resolution predictions act as soft targets for low-resolution predictions.
• Adapts an encoder-decoder architecture with a global context module and upsampling layer to improve UV map quality in the texture estimation network.
• Trains the model progressively across resolutions, starting from higher to lower resolutions, to stabilize optimization and improve convergence.

Experimental results

Research questions

• RQ1 Can a single deep neural network effectively estimate 3D human pose and shape across a wide range of low image resolutions without resolution-specific fine-tuning?
• RQ2 How can self-supervision be leveraged to improve 3D estimation when 3D annotations are sparse or unavailable?
• RQ3 Can contrastive learning improve feature representation consistency across resolutions more effectively than mean squared error loss?
• RQ4 To what extent does progressive training improve model stability and performance on multi-resolution low-resolution inputs?
• RQ5 Can the proposed method generalize to video inputs and produce high-quality textured 3D pedestrians from low-resolution sequences?

Key findings

• RSC-Net achieves state-of-the-art performance on low-resolution 3D human pose and shape estimation, outperforming existing SOTA methods on multiple benchmarks.
• The ablation study confirms that all components—resolution-aware network, self-supervision, and contrastive learning—contribute significantly to performance improvement.
• The directional self-supervision loss outperforms the original symmetric self-supervision loss by leveraging confidence in high-resolution predictions, reducing degradation of high-res outputs.
• The contrastive feature loss leads to better feature representations than MSE, as shown by improved performance in cross-resolution consistency and 3D estimation accuracy.
• The texture estimation network with global context and upsampling layers achieves higher-quality UV maps, with the point-wise summation fusion method outperforming concatenation.
• The method generalizes well to video inputs, enabling high-quality textured 3D pedestrian reconstruction from low-resolution video sequences.