[Paper Review] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields

Zhe Cao, Tomas Simon+2 more|arXiv (Cornell University)|Nov 24, 2016

Human Pose and Action Recognition32 references102 citations

TL;DR

The paper introduces Part Affinity Fields (PAFs), a bottom-up CNN that jointly predicts body part confidence maps and limb affinity fields to perform real-time multi-person 2D pose estimation with a greedy parsing stage. It achieves state-of-the-art accuracy on MPII and COCO benchmarks with efficient runtime independent of the number of people.

ABSTRACT

We present an approach to efficiently detect the 2D pose of multiple people in an image. The approach uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. The architecture encodes global context, allowing a greedy bottom-up parsing step that maintains high accuracy while achieving realtime performance, irrespective of the number of people in the image. The architecture is designed to jointly learn part locations and their association via two branches of the same sequential prediction process. Our method placed first in the inaugural COCO 2016 keypoints challenge, and significantly exceeds the previous state-of-the-art result on the MPII Multi-Person benchmark, both in performance and efficiency.

Motivation & Objective

• Detect 2D poses for multiple people in a single image without a person detector.
• Jointly learn body part locations and limb associations to preserve global context.
• Enable fast, scalable parsing that scales with image content rather than number of people.

Proposed method

• Two-branch CNN predicts confidence maps for body parts and Part Affinity Fields (PAFs) encoding limb orientation.
• Multi-stage architecture with iterative refinement and intermediate supervision for both branches.
• Greedy parsing using line integrals over PAFs to score candidate limbs and assemble full-body poses.
• Minimal spanning tree based parsing with bipartite matchings per limb type to form poses, improving efficiency over fully connected graphs.
• Loss functions with spatial masking to handle unlabeled regions and stage-wise supervision (L2 losses on S and L).
• Use of non-maximum suppression on confidence maps to generate part candidates before parsing.

Experimental results

Research questions

• RQ1 How can part detections and part-part associations be learned jointly to enable robust multi-person pose parsing?
• RQ2 Can a bottom-up approach with global limb affinity fields achieve real-time performance while maintaining high accuracy on challenging datasets?
• RQ3 What parsing strategy suffices to assemble correct poses from part candidates efficiently?
• RQ4 What is the impact of modelling limb orientation via PAFs vs simpler midpoint-based associations?
• RQ5 How does the method perform on MPII and COCO benchmarks relative to prior bottom-up and top-down approaches?

Key findings

• PAFs provide a robust, location-and-orientation representation for limb associations, enabling accurate multi-person parsing.
• The proposed two-branch, multi-stage CNN achieves state-of-the-art mAP on MPII and COCO benchmarks, outperforming previous bottom-up methods.
• Greedy, tree-structured parsing with PAFs is orders of magnitude faster than full graph-based optimizations while maintaining high accuracy.
• Runtime analysis shows the approach scales with image content rather than number of people, achieving 8.8 fps on a 19-person video in experiments.
• Training with masks for unlabeled persons improves accuracy by reducing penalties on missing annotations.