[Paper Review] Rethinking on Multi-Stage Networks for Human Pose Estimation

Wenbo Li, Zhicheng Wang+8 more|arXiv (Cornell University)|Jan 1, 2019

Human Pose and Action RecognitionComputer Science46 references109 citations

TL;DR

The paper revisits multi-stage pose estimation and shows that with a better single-stage module, cross-stage feature aggregation, and coarse-to-fine supervision, MSPN achieves state-of-the-art results on COCO and MPII, challenging the notion that multi-stage designs underperform single-stage ones.

ABSTRACT

Existing pose estimation approaches fall into two categories: single-stage and multi-stage methods. While multi-stage methods are seemingly more suited for the task, their performance in current practice is not as good as single-stage methods. This work studies this issue. We argue that the current multi-stage methods' unsatisfactory performance comes from the insufficiency in various design choices. We propose several improvements, including the single-stage module design, cross stage feature aggregation, and coarse-to-fine supervision. The resulting method establishes the new state-of-the-art on both MS COCO and MPII Human Pose dataset, justifying the effectiveness of a multi-stage architecture. The source code is publicly available for further research.

Motivation & Objective

• Assess why multi-stage networks underperform compared to single-stage ones on challenging pose datasets.
• Design improvements to unlock the potential of multi-stage architectures for pose estimation.
• Demonstrate the effectiveness of a refined single-stage module, cross-stage feature fusion, and coarse-to-fine supervision.
• Evaluate MSPN on COCO and MPII against prior methods and analyze component contributions.

Proposed method

• Adopt a two-stage top-down framework using detected human boxes as input to the MSPN.
• Replace the naive Hourglass-style single-stage module with a modern, high-capacity single-stage module (GlobalNet from CPN) for better feature processing.
• Introduce cross-stage feature aggregation to propagate multi-scale features across stages and strengthen information flow.
• Apply coarse-to-fine supervision with stage-specific Gaussian kernel sizes and multiple intermediate scales to improve localization progressively.
• Incorporate online hard key point mining (OHKM) on the largest scale in each stage and standard L2 losses across scales.

Experimental results

Research questions

• RQ1 Can a refined single-stage module improve the performance of multi-stage pose networks?
• RQ2 Does cross-stage feature aggregation reduce information loss in multi-stage architectures?
• RQ3 Does coarse-to-fine supervision improve keypoint localization accuracy over traditional multi-scale supervision?
• RQ4 What is MSPN's performance on COCO and MPII compared to prior state-of-the-art methods?
• RQ5 How sensitive is MSPN to detector quality and backbone choice?

Key findings

• MSPN with the proposed improvements significantly outperforms previous multi-stage methods and surpasses single-stage baselines of similar capacity.
• Using a ResNet-based GlobalNet as the single-stage module yields strong baseline performance and enables larger gains when combined with multi-stage design.
• Cross-stage feature aggregation provides measurable gains (e.g., MSPN improvement from 74.2 to 74.5 AP on COCO minival).
• Coarse-to-fine supervision substantially boosts localization accuracy and improves MSPN by notable margins across settings; it also benefits Hourglass when applied to a comparable FLOPs budget.
• MSPN achieves state-of-the-art results on COCO test-dev (76.1 AP single model; 77.1 AP with external data; 78.1 AP ensemble) and MPII (92.6 PCKh@0.5).
• Detector quality has limited impact on MSPN performance, indicating the gains mainly stem from architectural design.