[Paper Review] Convolutional Pose Machines

Evan Murray, von Coler, Henrik|arXiv (Cornell University)|Jan 30, 2016

Human Pose and Action Recognition37 references295 citations

TL;DR

Convolutional Pose Machines (CPMs) integrate deep convolutional networks into a sequential pose estimation framework to learn image features and implicit spatial models, producing progressively refined belief maps for body parts without explicit graphical-model inference. They achieve state-of-the-art results on MPII, LSP, and FLIC benchmarks.

ABSTRACT

Pose Machines provide a sequential prediction framework for learning rich implicit spatial models. In this work we show a systematic design for how convolutional networks can be incorporated into the pose machine framework for learning image features and image-dependent spatial models for the task of pose estimation. The contribution of this paper is to implicitly model long-range dependencies between variables in structured prediction tasks such as articulated pose estimation. We achieve this by designing a sequential architecture composed of convolutional networks that directly operate on belief maps from previous stages, producing increasingly refined estimates for part locations, without the need for explicit graphical model-style inference. Our approach addresses the characteristic difficulty of vanishing gradients during training by providing a natural learning objective function that enforces intermediate supervision, thereby replenishing back-propagated gradients and conditioning the learning procedure. We demonstrate state-of-the-art performance and outperform competing methods on standard benchmarks including the MPII, LSP, and FLIC datasets.

Motivation & Objective

• Motivate and design a system that learns implicit long-range spatial dependencies for articulated pose estimation.
• Replace hand-crafted features and graphical-model inference with a differentiable, end-to-end trainable convolutional architecture.
• Address vanishing gradients in deep sequential networks via intermediate supervision at each stage.
• Demonstrate state-of-the-art accuracy on standard pose benchmarks and analyze training schemes for CPMs.

Proposed method

• Replace pose-machine predictors with multi-stage convolutional networks that predict belief maps for each part at every stage.
• Each stage uses image evidence and previous stage belief maps to produce refined belief maps, enabling large receptive fields on both image and belief maps.
• Train end-to-end with intermediate supervision by adding an L2 loss on belief maps at every stage to combat vanishing gradients.
• Share image feature maps across stages and progressively increase the receptive field to capture long-range part relationships.
• Evaluate on MPII, LSP, and FLIC with data augmentation and multi-scale belief map fusion for final predictions.

Experimental results

Research questions

• RQ1 Can a fully differentiable, multi-stage convolutional architecture learn implicit spatial models for pose estimation without graphical model inference?
• RQ2 Does intermediate supervision effectively mitigate vanishing gradients in a deep, staged CNN for structured prediction?
• RQ3 How does increasing receptive field in later stages influence accuracy for long-range part dependencies?
• RQ4 What are the relative benefits of end-to-end joint training versus staged or non-supervised training in CPMs?
• RQ5 Do CPMs achieve state-of-the-art performance across MPII, LSP, and FLIC datasets on high- and low-precision metrics?

Key findings

• The CPM achieves state-of-the-art results on MPII, LSP, and FLIC datasets.
• Intermediate supervision mitigates vanishing gradients and improves learning across many layers.
• Larger receptive fields for later stages yield better modeling of long-range part interactions and improved accuracy.
• End-to-end training with joint optimization and intermediate losses significantly outperforms stage-wise or non-supervised training.
• Performance improves with up to five stages, with diminishing returns at the sixth stage in their experiments.