[Paper Review] OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields

Zhe Cao, Gines Hidalgo+3 more|arXiv (Cornell University)|Dec 18, 2018

Human Pose and Action Recognition69 references673 citations

TL;DR

OpenPose presents a real-time bottom-up 2D multi-person pose estimation method using Part Affinity Fields (PAFs) to associate body parts, and releases an open-source library for body, foot, hand, and facial keypoints.

ABSTRACT

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.

Motivation & Objective

• Motivate real-time understanding of people in images and videos through accurate multi-person 2D pose estimation.
• Address challenges of unknown person counts, occlusion, and runtime scaling with the number of people.
• Introduce Part Affinity Fields (PAFs) as a bottom-up representation to couple detection and association.
• Release OpenPose as an open-source system handling body, foot, hand, and facial keypoints.

Proposed method

• A CNN predicts confidence maps for body parts and PAFs encoding limb orientation.
• Multi-stage network with intermediate supervision enhances learning; PAF refinement is crucial while body-part refinement is less critical.
• Replace 7x7 convolutions with three 3x3 layers, concatenated to preserve receptive field and improve speed (DenseNet-like connections).
• Use greedy parsing over PAFs with line-integral limb scoring and bipartite matching to assemble body poses for multiple people.
• Extend to foot keypoints with a public foot dataset and demonstrate combined body+foot detection without sacrificing speed or accuracy.

Experimental results

Research questions

• RQ1 Can Part Affinity Fields enable accurate, real-time, bottom-up multi-person pose parsing without heavy dependence on person detectors?
• RQ2 How does refining PAFs versus body part confidence maps affect accuracy and speed in multi-person parsing?
• RQ3 Does combining body and foot keypoint detection improve pose estimation performance and efficiency?
• RQ4 How does OpenPose compare to existing methods (e.g., Mask R-CNN, Alpha-Pose) in runtime and accuracy on standard benchmarks?

Key findings

• Achieves real-time multi-person 2D pose estimation with competitive accuracy across benchmarks.
• PAF refinement is crucial for accuracy, while refining body part confidence maps is less beneficial.
• Network depth increased with faster, more accurate results (~200% speed and ~7% accuracy gain reported in sections).
• Introduces an annotated foot dataset and shows combined body+foot keypoint detector maintains body accuracy while reducing inference time.
• OpenPose is the first open-source real-time system for body, foot, hand, and facial keypoints (up to 135 keypoints) and runs around 22 FPS on a GTX 1080 Ti.