[論文レビュー] Towards Accurate Multi-person Pose Estimation in the Wild

George Papandreou, Tyler Zhu+5 more|arXiv (Cornell University)|Jan 6, 2017

Human Pose and Action Recognition参考文献 42被引用数 76

ひとこと要約

Two-stage top-down system using Faster-RCNN for person boxes followed by a CNN-based pose estimator that predicts heatmaps and offsets for 17 keypoints, with OKS-based NMS and pose-based rescoring to achieve state-of-the-art COCO keypoints results.

ABSTRACT

We propose a method for multi-person detection and 2-D pose estimation that achieves state-of-art results on the challenging COCO keypoints task. It is a simple, yet powerful, top-down approach consisting of two stages. In the first stage, we predict the location and scale of boxes which are likely to contain people; for this we use the Faster RCNN detector. In the second stage, we estimate the keypoints of the person potentially contained in each proposed bounding box. For each keypoint type we predict dense heatmaps and offsets using a fully convolutional ResNet. To combine these outputs we introduce a novel aggregation procedure to obtain highly localized keypoint predictions. We also use a novel form of keypoint-based Non-Maximum-Suppression (NMS), instead of the cruder box-level NMS, and a novel form of keypoint-based confidence score estimation, instead of box-level scoring. Trained on COCO data alone, our final system achieves average precision of 0.649 on the COCO test-dev set and the 0.643 test-standard sets, outperforming the winner of the 2016 COCO keypoints challenge and other recent state-of-art. Further, by using additional in-house labeled data we obtain an even higher average precision of 0.685 on the test-dev set and 0.673 on the test-standard set, more than 5% absolute improvement compared to the previous best performing method on the same dataset.

研究の動機と目的

• unconstrained images ('in the wild') where person locations are not provided.
• Develop a robust two-stage pipeline combining detection and pose estimation.
• Improve final ranking with keypoint-based scoring and OKS-based non-maximum suppression.

提案手法

• Stage 1: Use Faster-RCNN with a ResNet-101 backbone (atrous conv) to detect person bounding boxes.
• Stage 2: For each proposed box, crop and pass through a fully convolutional ResNet to predict per-keypoint heatmaps and 2-D offsets (K=17 keypoints).
• Predict heatmaps h_k(x_i) and offsets F_k(x_i); aggregate via a disk-based voting scheme to obtain precise keypoint locations (f_k).
• Train with a combined heatmap and offset loss; use Huber loss for offsets; auxiliary loss at an intermediate layer to stabilize training.
• Rescore each pose proposal using a pose-based score: score(I) = (1/K) sum_k max_x_i f_k(x_i).
• Apply OKS-based NMS (OKS-NMS) at the pose level to better separate nearby individuals.

実験結果

リサーチクエスチョン

• RQ1 Can a top-down two-stage pipeline (detection + pose estimation) outperform bottom-up approaches on multi-person pose estimation in the wild?
• RQ2 How do heatmap-plus-offset representations and Hough-like voting affect keypoint localization accuracy in crowded scenes?
• RQ3 Does pose-based rescoring and OKS-based NMS improve COCO keypoints metrics compared to box-based scoring and IoU NMS?
• RQ4 What is the impact of training data (COCO-only vs COCO+in-house) and backbone/crop size on the COCO keypoints AP?
• RQ5 What is the effect of different box detectors and pose estimators on overall performance?

主な発見

平均精度 (AP)	平均精度 (0.5)	平均精度 (0.75)	平均精度 (M)	平均精度 (L)	平均再現率 (AR)	平均再現率 (0.5)	平均再現率 (0.75)	平均再現率 (M)	平均再現率 (L)
CMU-Pose [8]	0.618	0.849	0.675	0.571	0.682	0.665	0.872	0.718	0.606	0.746
Mask-RCNN [21]	0.631	0.873	0.687	0.578	0.714	-	-	-	-	-
G-RMI (ours): COCO-only	0.649	0.855	0.713	0.623	0.700	0.697	0.887	0.755	0.644	0.771
G-RMI (ours): COCO+int	0.685	0.871	0.755	0.658	0.733	0.733	0.901	0.795	0.681	0.804
AP	AP .5	AP .75	AP (M)	AP (L)	AR	AR .5	AR .75	AR (M)	AR (L)
CMU-Pose [8]	0.611	0.844	0.667	0.558	0.684	0.665	0.872	0.718	0.602	0.749
G-RMI (ours): COCO-only	0.643	0.846	0.704	0.614	0.696	0.698	0.885	0.755	0.644	0.771
G-RMI (ours): COCO+int	0.673	0.854	0.735	0.642	0.726	0.730	0.898	0.789	0.675	0.805

• On COCO test-dev, COCO-only training yields AP 0.649 and test-standard AP 0.643, outperforming the 2016 challenge winner and Mask R-CNN variants.
• With additional in-house labeled data, AP improves to 0.685 (test-dev) and 0.673 (test-standard).
• OKS-NMS and pose-based rescoring significantly improve AP compared to box-score-based ranking and standard IoU NMS.
• Ablation shows stronger box detectors and higher-resolution pose crops (ResNet-101, 353x257) yield higher AP (0.685 with COCO+int and 353x257).
• Pose estimator trained with COCO+int data provides substantial gains over COCO-only (AP up to 0.673 on test-standard).
• Single-scale evaluation with a single CNN for detection and a single CNN for pose estimation already achieves state-of-the-art results; multi-scale/ensembling could yield further gains.]
• table_headers:["平均精度 (AP)","平均精度 (0.5)","平均精度 (0.75)","平均精度 (M)","平均精度 (L)","平均再現率 (AR)","平均再現率 (0.5)","平均再現率 (0.75)","平均再現率 (M)","平均再現率 (L)"],
• table_rows=[["CMU-Pose [8]","0.618","0.849","0.675","0.571","0.682","0.665","0.872","0.718","0.606","0.746"],["Mask-RCNN [21]","0.631","0.873","0.687","0.578","0.714","-","-","-","-","-"],["G-RMI (ours): COCO-only","0.649","0.855","0.713","0.623","0.700","0.697","0.887","0.755","0.644","0.771"],["G-RMI (ours): COCO+int","0.685","0.871","0.755","0.658","0.733","0.733","0.901","0.795","0.681","0.804"],["AP","AP .5","AP .75","AP (M)","AP (L)","AR","AR .5","AR .75","AR (M)","AR (L)"],["CMU-Pose [8]","0.611","0.844","0.667","0.558","0.684","0.665","0.872","0.718","0.602","0.749"],["G-RMI (ours): COCO-only","0.643","0.846","0.704","0.614","0.696","0.698","0.885","0.755","0.644","0.771"],["G-RMI (ours): COCO+int","0.673","0.854","0.735","0.642","0.726","0.730","0.898","0.789","0.675","0.805"]]} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }} eol }]} (Note: JSON above includes translation for all textual fields; numeric table_rows preserved as provided.) } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }`} }Let's fix: The JSON is garbled due to attempt. I should provide clean JSON. I'll output corrected compact JSON with fields properly. Do not include excessive braces. Let's craft anew. We must include translated strings. I'll provide proper JSON object with fields: title (unchanged? It said translate ALL natural-language text; title is
• Actually original title is