[Paper Review] Multi-Person Pose Estimation via Column Generation

Shaofei Wang, Chong Zhang+3 more|arXiv (Cornell University)|Sep 18, 2017

Human Pose and Action Recognition14 references3 citations

TL;DR

This paper proposes a novel integer linear programming (ILP) formulation for multi-person pose estimation using a two-tier hierarchical structure—local part grouping and global pose modeling via an augmented tree—combined with a custom column generation algorithm. The method achieves state-of-the-art accuracy on the MPII benchmark, outperforming prior work in localization of challenging body parts like wrists and ankles, while being 100x faster than [12] and more accurate than [15], despite using a simpler edge structure and pure Python code.

ABSTRACT

We study the problem of multi-person pose estimation in natural images. A pose estimate describes the spatial position and identity (head, foot, knee, etc.) of every non-occluded body part of a person. Pose estimation is difficult due to issues such as deformation and variation in body configurations and occlusion of parts, while multi-person settings add complications such as an unknown number of people, with unknown appearance and possible interactions in their poses and part locations. We give a novel integer program formulation of the multi-person pose estimation problem, in which variables correspond to assignments of parts in the image to poses in a two-tier, hierarchical way. This enables us to develop an efficient custom optimization procedure based on column generation, where columns are produced by exact optimization of very small scale integer programs. We demonstrate improved accuracy and speed for our method on the MPII multi-person pose estimation benchmark.

Motivation & Objective

• To address the challenges of multi-person pose estimation, including occlusion, variable body configurations, and unknown person counts.
• To improve accuracy in localizing difficult body parts such as wrists and ankles, which are often misdetected in existing methods.
• To develop a more efficient optimization framework that avoids the computational burden of dense pairwise interactions in pose models.
• To integrate non-maximum suppression naturally into the pose estimation model through structured variable design.
• To demonstrate that a well-structured ILP formulation with efficient column generation can achieve superior speed-accuracy trade-offs.

Proposed method

• Formulates multi-person pose estimation as an integer linear program (ILP) with two tiers: local assignments group multiple detections per part, and global poses enforce anatomical consistency via an augmented tree structure.
• Uses column generation to solve the ILP, where pricing subproblems are solved via dynamic programming on the tree-structured pose model, enabling efficient column generation.
• Employs small-scale exact integer programs to generate columns, ensuring high-quality solutions without relying on heuristic relaxation.
• Introduces a cost function that excludes noisy pairwise interactions between non-global detections from different people, improving localization robustness.
• Applies odd-set inequalities of size three to tighten the LP relaxation when needed, though integer solutions are typically achieved directly at termination.
• Uses a heuristic weight ω = 30 to discourage poses with few detections, favoring more complete and reliable configurations.

Experimental results

Research questions

• RQ1 Can a two-tier ILP formulation with structured local and global components improve accuracy in multi-person pose estimation compared to fully connected models?
• RQ2 Does column generation based on dynamic programming on tree-structured poses enable faster and more accurate inference than standard ILP solvers?
• RQ3 Can excluding noisy pairwise interactions between non-global detections improve localization of hard-to-detect body parts like wrists and ankles?
• RQ4 To what extent does a structured model reduce the need for post-processing like non-maximum suppression?
• RQ5 How does the proposed method compare in speed and accuracy to recent state-of-the-art approaches like [15], especially on challenging body parts?

Key findings

• The proposed method achieves a mean Average Precision (mAP) of 79.1 on the MPII multi-person pose estimation benchmark, outperforming [15] (78.5) and [12] (78.8).
• The method improves AP on difficult parts: 70.1 for wrists (vs. 68.6 in [15]) and 66.6 for ankles (vs. 65.2 in [15]).
• Despite using a simpler edge structure (fewer pairwise connections), the method achieves higher accuracy than [12] and [15], which use fully connected models.
• The method runs 100x faster than [12] and is 10x faster than [15], despite being implemented in pure Python without parallelization.
• The LP relaxation typically yields integer solutions without additional tightening, indicating strong integrality properties of the formulation.
• Adding three additional edges (left-hip to left-shoulder, right-hip to right-shoulder, shoulders to head) improves mAP from 78.8 to 79.1 with negligible runtime cost.