[Paper Review] DirectPose: Direct End-to-End Multi-Person Pose Estimation

Zhi Tian, Hao Chen+1 more|arXiv (Cornell University)|Nov 18, 2019

Human Pose and Action RecognitionComputer Science29 references80 citations

TL;DR

DirectPose provides a fully end-to-end single-shot framework for direct instance-aware keypoint prediction without bounding-box detection or post-hoc grouping, aided by a novel Keypoint Alignment (KPAlign) module and optional heatmap-based regularization during training.

ABSTRACT

We propose the first direct end-to-end multi-person pose estimation framework, termed DirectPose. Inspired by recent anchor-free object detectors, which directly regress the two corners of target bounding-boxes, the proposed framework directly predicts instance-aware keypoints for all the instances from a raw input image, eliminating the need for heuristic grouping in bottom-up methods or bounding-box detection and RoI operations in top-down ones. We also propose a novel Keypoint Alignment (KPAlign) mechanism, which overcomes the main difficulty: lack of the alignment between the convolutional features and predictions in this end-to-end framework. KPAlign improves the framework's performance by a large margin while still keeping the framework end-to-end trainable. With the only postprocessing non-maximum suppression (NMS), our proposed framework can detect multi-person keypoints with or without bounding-boxes in a single shot. Experiments demonstrate that the end-to-end paradigm can achieve competitive or better performance than previous strong baselines, in both bottom-up and top-down methods. We hope that our end-to-end approach can provide a new perspective for the human pose estimation task.

Motivation & Objective

• Motivate a direct end-to-end approach for multi-person pose estimation that bypasses bounding-box detection and keypoint grouping.
• Eliminate non-differentiable post-processing by introducing an end-to-end trainable pipeline.
• Improve keypoint localization accuracy through feature-prediction alignment (KPAlign).
• Demonstrate competitiveness with strong top-down and bottom-up baselines on COCO.

Proposed method

• Extend the anchor-free FCOS detector with a keypoint detection head to regress 2K coordinates for K keypoints per instance.
• Introduce KPAlign to align local features with predicted keypoints via a locator (sampling locations) and a predictor (per-keypoint regression).
• Allow end-to-end regression-based keypoint prediction with differentiable sampling and alignment.
• Optional heatmap-based auxiliary task during training to regularize regression learning (removed at test).
• Experiment with grouping and separate feature maps to reduce computation and improve performance.
• Evaluate with and without bounding-box detection, and compare to state-of-the-art top-down and bottom-up methods on COCO.

Experimental results

Research questions

• RQ1 Can an end-to-end single-stage framework directly map an input image to instance-aware keypoints without bounding-box detection or RoI-based operations?
• RQ2 Does a feature-keypoint alignment (KPAlign) significantly improve end-to-end keypoint regression performance?
• RQ3 How does the end-to-end approach compare to traditional top-down and bottom-up methods on COCO in terms of accuracy and speed?
• RQ4 What is the impact of auxiliary heatmap learning on the regression-based keypoint predictions during training?
• RQ5 Is the method robust when optionally combined with bounding-box detection for shared tasks?

Key findings

• End-to-end DirectPose with KPAlign achieves competitive keypoint AP on COCO compared to strong baselines.
• KPAlign provides a large performance boost over naive end-to-end keypoint regression (over 7 AP points in most ablations).
• Grouped KPAlign and separate feature maps further improve accuracy with modest computational trade-offs.
• Joint heatmap learning as an auxiliary training task significantly improves regression-based keypoint AP (e.g., from 52.2 to 58.0 AP with 8x heatmaps).
• Without bells-and-whistles, DirectPose (R-50) achieves 62.2 AP kp on COCO test-dev; with R-101, 63.3 AP kp; multi-scale testing raises to 63.0 and 64.8 respectively.
• The method runs with around 74-87 ms per image on COCO minival with ResNet backbones, comparable to or faster than Mask R-CNN under similar settings.
• When combined with bounding-box detection, the framework can achieve 61.5 AP kp and 55.3 AP bb on minival, showing compatibility with a bounding-box branch.