[Paper Review] Stacked Hourglass Networks for Human Pose Estimation

Alejandro Newell, Kaiyu Yang+1 more|arXiv (Cornell University)|Mar 22, 2016

Human Pose and Action Recognition41 references536 citations

TL;DR

Introduces stacked hourglass convolutional networks with intermediate supervision to predict human pose heatmaps, achieving state-of-the-art results on FLIC and MPII benchmarks.

ABSTRACT

This work introduces a novel convolutional network architecture for the task of human pose estimation. Features are processed across all scales and consolidated to best capture the various spatial relationships associated with the body. We show how repeated bottom-up, top-down processing used in conjunction with intermediate supervision is critical to improving the performance of the network. We refer to the architecture as a "stacked hourglass" network based on the successive steps of pooling and upsampling that are done to produce a final set of predictions. State-of-the-art results are achieved on the FLIC and MPII benchmarks outcompeting all recent methods.

Motivation & Objective

• Develop a convolutional architecture that captures and consolidates information across multiple scales for precise keypoint localization.
• Demonstrate that repeated bottom-up, top-down inference through stacked hourglass modules improves pose estimation.
• Show the benefits of intermediate supervision in training multi-stage pose prediction networks.
• Achieve state-of-the-art performance on standard pose benchmarks (FLIC and MPII).

Proposed method

• Propose a symmetric hourglass module that processes features from high to low resolution and back, using nearest-neighbor upsampling and skip connections to fuse multi-scale information.
• Stack multiple hourglass modules end-to-end without shared weights to enable repeated bottom-up, top-down refinement.
• Incorporate intermediate supervision by producing heatmaps after each hourglass and applying losses to guide training.
• Use residual modules within each hourglass to control parameter count and enable deeper architectures with bottleneck designs.
• Train with mean-squared error heatmap loss against ground-truth Gaussian heatmaps, with data augmentation and batch normalization.
• Evaluate on FLIC and MPII using PCK and PCKh metrics, and apply test-time image flipping for predictions."

Experimental results

Research questions

• RQ1 Does stacking multiple hourglass modules improve pose estimation accuracy beyond a single hourglass with equivalent parameter count?
• RQ2 What is the impact of intermediate supervision on training dynamics and final accuracy?
• RQ3 How does the proposed network handle challenges such as occlusion and close-proximity multiple people in practice?
• RQ4 Can a purely convolutional, heatmap-based approach achieve state-of-the-art results without graphical models or explicit body priors?

Key findings

• The eight-hourglass stacked network with intermediate supervision achieves state-of-the-art results on MPII, with notable improvements on difficult joints (elbows and wrists).
• On FLIC, the model reaches 99.0% PCK@0.2 for elbows and 97.0% for wrists.
• The final model achieved per-joint ACC of Head 98.2, Shoulder 96.3, Elbow 91.2, Wrist 87.1, Hip 90.1, Knee 87.4, Ankle 83.6, Total 90.9 on MPII.
• Stacking hourglasses provides consistent accuracy gains over fewer stacks, and intermediate supervision yields additional gains when combined with stacking.
• The method can predict poses for a single person in images with occlusion and close-proximity figures, though occlusion remains a significant challenge.
• Predictions are generated as heatmaps with post-processing that selects the maximum activation per joint; test-time augmentation includes flipping the image and averaging heatmaps.