[Paper Review] Human Pose Regression by Combining Indirect Part Detection and Contextual Information

Diogo Luvizon, Hedi Tabia+1 more|arXiv (Cornell University)|Oct 6, 2017

Human Pose and Action RecognitionComputer Science46 references235 citations

TL;DR

An end-to-end trainable pose regression method using a differentiable Soft-argmax to convert heat maps to joint coordinates, augmented with contextual information, achieving competitive results with regression methods and close to detection-based approaches.

ABSTRACT

In this paper, we propose an end-to-end trainable regression approach for human pose estimation from still images. We use the proposed Soft-argmax function to convert feature maps directly to joint coordinates, resulting in a fully differentiable framework. Our method is able to learn heat maps representations indirectly, without additional steps of artificial ground truth generation. Consequently, contextual information can be included to the pose predictions in a seamless way. We evaluated our method on two very challenging datasets, the Leeds Sports Poses (LSP) and the MPII Human Pose datasets, reaching the best performance among all the existing regression methods and comparable results to the state-of-the-art detection based approaches.

Motivation & Objective

• Motivate bridging the gap between regression and detection in 2D human pose estimation.
• Propose a differentiable Soft-argmax layer to directly regresses 2D joint coordinates from heat maps.
• Incorporate part-based and contextual heat maps to improve pose predictions.
• Enable end-to-end training without artificially generated heat maps during training.
• Demonstrate competitive accuracy against detection-based methods on standard benchmarks.

Proposed method

• Introduce a CNN architecture with Stem, Block-A, and Block-B producing part-based and context heat maps.
• Use Soft-argmax to convert heat maps into joint coordinates in a fully differentiable way.
• Predict joint probabilities via a sigmoid on global max-pooling of each heat map.
• Aggregate part-based and context-derived predictions through a weighted scheme controlled by joint-context probabilities.
• Train with elastic net loss for joint coordinates and binary cross-entropy for joint presence probabilities.
• Employ data augmentation and train on MPII with fine-tuning on LSP, using multiple prediction blocks for intermediate supervision.

Experimental results

Research questions

• RQ1 Can a differentiable Soft-argmax allow end-to-end training from RGB images to 2D joint coordinates without artificial heat-map ground truth?
• RQ2 Does incorporating contextual heat maps alongside part-based maps improve regression-based pose estimation performance?
• RQ3 How close can a regression-based approach get to state-of-the-art detection-based methods on challenging benchmarks like MPII and LSP?
• RQ4 What is the effect of intermediate supervision and multi-block prediction on training stability and accuracy?
• RQ5 How effective is joint probability as a learned indicator of joint visibility in improving pose estimates?

Key findings

• Achieves competitive regression-based pose accuracy close to state-of-the-art detection methods on MPII and surpasses other regression methods.
• Soft-argmax provides sub-pixel precision and differentiability enabling end-to-end learning.
• Context maps and joint probabilities improve pose precision, particularly via contextual aggregation.
• Outperforms several regression-based methods on LSP under OC annotations across PCK and PCP metrics.
• On MPII, reaches 91.2% test score (Single Person challenge), just 0.7% below the best detection-based method.
• The model uses a smaller network with comparable performance to heavier architectures.