[Paper Review] DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model

Eldar Insafutdinov, Leonid Pishchulin+3 more|arXiv (Cornell University)|May 10, 2016

Human Pose and Action Recognition12 references115 citations

TL;DR

DeeperCut advances multi-person pose estimation by 1) deep, strong body-part detectors; 2) image-conditioned pairwise terms for assembling parts; and 3) an incremental optimization strategy that dramatically speeds up inference while improving accuracy.

ABSTRACT

The goal of this paper is to advance the state-of-the-art of articulated pose estimation in scenes with multiple people. To that end we contribute on three fronts. We propose (1) improved body part detectors that generate effective bottom-up proposals for body parts; (2) novel image-conditioned pairwise terms that allow to assemble the proposals into a variable number of consistent body part configurations; and (3) an incremental optimization strategy that explores the search space more efficiently thus leading both to better performance and significant speed-up factors. Evaluation is done on two single-person and two multi-person pose estimation benchmarks. The proposed approach significantly outperforms best known multi-person pose estimation results while demonstrating competitive performance on the task of single person pose estimation. Models and code available at http://pose.mpi-inf.mpg.de

Motivation & Objective

• Improve body part detection with deep learning to generate high-quality bottom-up proposals.
• Introduce image-conditioned pairwise terms to correctly assemble body parts into poses in crowded scenes.
• Develop an incremental optimization strategy to significantly speed up inference without sacrificing accuracy.
• Demonstrate state-of-the-art performance on single-person and multi-person pose benchmarks.

Proposed method

• Use a very deep residual network (ResNet) based part detector with a fully convolutional architecture to produce scoremaps for body parts.
• Adapt ResNet to maintain a fine-grained 8 px stride and employ deconvolution/holes to recover spatial resolution for part localization.
• Incorporate intermediate supervision by adding part loss layers inside the conv4 block to improve gradient flow and spatial disambiguation.
• Train a image-conditioned pairwise terms model that regresses from each part location to relative positions of other joints, producing features to compute pairwise costs via a logistic model p(z=1|f, ω).
• Compute pairwise costs by comparing CNN-predicted offsets with actual inter-part offsets, including forward and backward orientations and angle terms.
• Optimize the overall body part selection and clustering into distinct people via an incremental branch-and-cut ILP solver that solves multiple smaller instances sequentially.

Experimental results

Research questions

• RQ1 How do deeper part detectors affect single- and multi-person pose estimation performance?
• RQ2 Can image-conditioned pairwise terms improve the grouping of body-part hypotheses into coherent multi-person poses in crowded scenes?
• RQ3 Does an incremental optimization strategy reduce runtime while maintaining or improving pose accuracy in multi-person settings?

Key findings

• Part detectors based on very deep ResNets achieve state-of-the-art PCK/AUC on LSP and MPII benchmarks, with intermediate supervision providing further gains.
• Image-conditioned pairwise terms significantly improve multi-person pose AP and reduce runtime dramatically (e.g., from 259,220 s/frame to 1,987 s/frame in one comparison).
• Bi-directional pairwise terms with angle features yield the best AP (52.6% AP) and lowest run-time (578 s/frame) in ablation studies.
• Incremental optimization (3-stage) raises AP to 57.6% and reduces median runtime to 271 s/frame, compared with the single-stage baseline.
• DeeperCut outperforms the baseline DeepCut and strong two-stage baselines while achieving run-time reductions by orders of magnitude.
• On MPII Multi-Person, DeeperCut with incremental optimization reaches 69.7% AP on subset data and 59.4% AP on full data, with substantial runtime savings.