[Paper Review] Deep High-Resolution Representation Learning for Human Pose Estimation

Ke Sun, Bin Xiao+2 more|arXiv (Cornell University)|Feb 25, 2019

Human Pose and Action Recognition72 references57 citations

TL;DR

This paper introduces HRNet, a high-resolution network that maintains high-resolution representations throughout processing and repeatedly fuses multi-scale features, achieving state-of-the-art pose estimation on COCO, MPII, and PoseTrack datasets.

ABSTRACT

This is an official pytorch implementation of Deep High-Resolution Representation Learning for Human Pose Estimation. In this work, we are interested in the human pose estimation problem with a focus on learning reliable high-resolution representations. Most existing methods recover high-resolution representations from low-resolution representations produced by a high-to-low resolution network. Instead, our proposed network maintains high-resolution representations through the whole process. We start from a high-resolution subnetwork as the first stage, gradually add high-to-low resolution subnetworks one by one to form more stages, and connect the mutli-resolution subnetworks in parallel. We conduct repeated multi-scale fusions such that each of the high-to-low resolution representations receives information from other parallel representations over and over, leading to rich high-resolution representations. As a result, the predicted keypoint heatmap is potentially more accurate and spatially more precise. We empirically demonstrate the effectiveness of our network through the superior pose estimation results over two benchmark datasets: the COCO keypoint detection dataset and the MPII Human Pose dataset. The code and models have been publicly available at \url{https://github.com/leoxiaobin/deep-high-resolution-net.pytorch}.

Motivation & Objective

• Motivate learning reliable and precise high-resolution representations for human pose estimation.
• Design a network that maintains high-resolution representations throughout all stages instead of recovering resolution from low-resolution features.
• Propose repeated multi-scale fusion across parallel high-to-low subnetworks to enrich high-resolution representations.
• Demonstrate superior keypoint heatmap accuracy on COCO and MPII, and improved pose tracking on PoseTrack.

Proposed method

• Propose High-Resolution Net (HRNet) that starts with a high-resolution subnetwork and gradually adds parallel high-to-low subnetworks.
• Connect multi-resolution subnetworks in parallel and perform repeated multi-scale fusion through exchange units across and within stages.
• Regress K heatmaps from the final high-resolution representations using a mean squared error loss against Gaussian-groundtruth heatmaps.
• Instantiate HRNet in small (W32) and large (W48) widths, with four stages and eight exchange units.
• Train with standard data augmentation, Adam optimizer, and imageNet-pretrained backbones for improved performance.

Experimental results

Research questions

• RQ1 Can maintaining high-resolution representations throughout the network improve keypoint localization accuracy compared to traditional high-to-low pipelines?
• RQ2 Does repeated multi-scale fusion across parallel subnetworks lead to richer high-resolution features and better heatmaps?
• RQ3 What are the performance gains of HRNet on COCO, MPII, and PoseTrack benchmarks relative to state-of-the-art methods?
• RQ4 How do network width and input resolution affect pose estimation accuracy and efficiency?
• RQ5 Is HRNet effective for video-based pose tracking beyond single-image pose estimation?

Key findings

• HRNet-W32 (no pretraining) achieves 73.4 AP on COCO val with 256x192 input, outperforming Hourglass with similar size and lower GFLOPs.
• HRNet-W32 (with pretraining) achieves 74.4 AP on COCO val, AP50 90.5, AP75 81.9, AR 79.8, outperforming the non-pretrained variant.
• HRNet-W48 (with pretraining) achieves 75.1 AP on COCO val, AP50 90.6, AP75 82.2, AR 80.4, demonstrating that increased width yields higher accuracy.
• On COCO test-dev, HRNet-W32 and HRNet-W48 achieve 74.9 AP and 75.5 AP respectively (single-model, top-down approach).
• On MPII, HRNet-W32 achieves 92.3 PCKh@0.5, surpassing several prior methods and matching the state of the art.
• For PoseTrack 2017, HRNet-W48 yields 74.9 mAP and 57.9 MOTA, outperforming several baselines and demonstrating strong video tracking performance.