[Paper Review] Towards Accurate Multi-person Pose Estimation in the Wild

George Papandreou, Tyler Zhu+5 more|arXiv (Cornell University)|Jan 6, 2017

Human Pose and Action RecognitionComputer Science42 references76 citations

TL;DR

A two-stage top-down system using Faster-RCNN for person boxes followed by a CNN-based pose estimator that predicts heatmaps and offsets for 17 keypoints, with OKS-based NMS and pose-based rescoring to achieve state-of-the-art COCO keypoints results.

ABSTRACT

We propose a method for multi-person detection and 2-D pose estimation that achieves state-of-art results on the challenging COCO keypoints task. It is a simple, yet powerful, top-down approach consisting of two stages. In the first stage, we predict the location and scale of boxes which are likely to contain people; for this we use the Faster RCNN detector. In the second stage, we estimate the keypoints of the person potentially contained in each proposed bounding box. For each keypoint type we predict dense heatmaps and offsets using a fully convolutional ResNet. To combine these outputs we introduce a novel aggregation procedure to obtain highly localized keypoint predictions. We also use a novel form of keypoint-based Non-Maximum-Suppression (NMS), instead of the cruder box-level NMS, and a novel form of keypoint-based confidence score estimation, instead of box-level scoring. Trained on COCO data alone, our final system achieves average precision of 0.649 on the COCO test-dev set and the 0.643 test-standard sets, outperforming the winner of the 2016 COCO keypoints challenge and other recent state-of-art. Further, by using additional in-house labeled data we obtain an even higher average precision of 0.685 on the test-dev set and 0.673 on the test-standard set, more than 5% absolute improvement compared to the previous best performing method on the same dataset.

Motivation & Objective

• Address multi-person pose estimation in unconstrained images ('in the wild') where person locations are not provided.
• Develop a robust two-stage pipeline combining detection and pose estimation.
• Improve final ranking with keypoint-based scoring and OKS-based non-maximum suppression.

Proposed method

• Stage 1: Use Faster-RCNN with a ResNet-101 backbone (atrous conv) to detect person bounding boxes.
• Stage 2: For each proposed box, crop and pass through a fully convolutional ResNet to predict per-keypoint heatmaps and 2-D offsets (K=17 keypoints).
• Predict heatmaps h_k(x_i) and offsets F_k(x_i); aggregate via a disk-based voting scheme to obtain precise keypoint locations (f_k).
• Train with a combined heatmap and offset loss; use Huber loss for offsets; auxiliary loss at an intermediate layer to stabilize training.
• Rescore each pose proposal using a pose-based score: score(I) = (1/K) sum_k max_x_i f_k(x_i).
• Apply OKS-based NMS (OKS-NMS) at the pose level to better separate nearby individuals.

Experimental results

Research questions

• RQ1 Can a top-down two-stage pipeline (detection + pose estimation) outperform bottom-up approaches on multi-person pose estimation in the wild?
• RQ2 How do heatmap-plus-offset representations and Hough-like voting affect keypoint localization accuracy in crowded scenes?
• RQ3 Does pose-based rescoring and OKS-based NMS improve COCO keypoints metrics compared to box-based scoring and IoU NMS?
• RQ4 What is the impact of training data (COCO-only vs COCO+in-house) and backbone/crop size on the COCO keypoints AP?
• RQ5 What is the effect of different box detectors and pose estimators on overall performance?

Key findings

• On COCO test-dev, COCO-only training yields AP 0.649 and test-standard AP 0.643, outperforming the 2016 challenge winner and Mask R-CNN variants.
• With additional in-house labeled data, AP improves to 0.685 (test-dev) and 0.673 (test-standard).
• OKS-NMS and pose-based rescoring significantly improve AP compared to box-score-based ranking and standard IoU NMS.
• Ablation shows stronger box detectors and higher-resolution pose crops (ResNet-101, 353x257) yield higher AP (0.685 with COCO+int and 353x257).
• Pose estimator trained with COCO+int data provides substantial gains over COCO-only (AP up to 0.673 on test-standard).
• Single-scale evaluation with a single CNN for detection and a single CNN for pose estimation already achieves state-of-the-art results; multi-scale/ensembling could yield further gains.