[Paper Review] Human Pose Estimation with Spatial Contextual Information

Hong Zhang, Hao Ouyang+5 more|arXiv (Cornell University)|Jan 7, 2019

Human Pose and Action RecognitionComputer Science54 references63 citations

TL;DR

The paper introduces Cascade Prediction Fusion (CPF) and Pose Graph Neural Network (PGNN) to leverage spatial contextual information for human pose estimation, achieving state-of-the-art results on MPII and LSP with minimal extra computation.

ABSTRACT

We explore the importance of spatial contextual information in human pose estimation. Most state-of-the-art pose networks are trained in a multi-stage manner and produce several auxiliary predictions for deep supervision. With this principle, we present two conceptually simple and yet computational efficient modules, namely Cascade Prediction Fusion (CPF) and Pose Graph Neural Network (PGNN), to exploit underlying contextual information. Cascade prediction fusion accumulates prediction maps from previous stages to extract informative signals. The resulting maps also function as a prior to guide prediction at following stages. To promote spatial correlation among joints, our PGNN learns a structured representation of human pose as a graph. Direct message passing between different joints is enabled and spatial relation is captured. These two modules require very limited computational complexity. Experimental results demonstrate that our method consistently outperforms previous methods on MPII and LSP benchmark.

Motivation & Objective

• Motivate the importance of spatial contextual information in pose estimation.
• Propose lightweight modules to leverage auxiliary stage predictions and joint relationships.
• Demonstrate end-to-end trainability of CPF and PGNN within common pose networks.
• Show improved accuracy on MPII and LSP with limited computational overhead.

Proposed method

• Cascade Prediction Fusion (CPF): progressively fuse auxiliary predictions from earlier stages as priors to guide later-stage predictions.
• PGNN (Pose Graph Neural Network): model joints as graph nodes with edge-based message passing to capture spatial relations.
• Graph construction: nodes for joints; edges encode neighboring joint relations; supports tree and loopy graph structures.
• Update mechanism: parallel node updates using a gated GRU-like update (no shared edge weights); combines new messages with memory.
• Training: end-to-end via L2 loss on predicted maps; uses BPTT for temporal extension risk, with an emphasis on spatial structure learning.
• Backbone compatibility: applies CPF and PGNN to ResNet-50 (modified with FPN and dilated convs) and Hourglass backbones.

Experimental results

Research questions

• RQ1 How does incorporating spatial contextual information via CPF improve refinement across pose estimation stages?
• RQ2 Can a graph-based neural network (PGNN) effectively model and propagate spatial relations between joints to improve localization?
• RQ3 Do CPF and PGNN provide consistent performance gains across different backbone networks on MPII and LSP?

Key findings

• CPF yields measurable improvements by using prior stage predictions to inform subsequent stage predictions.
• PGNN provides additional gains by enabling direct message passing between joints and learning structured spatial relations.
• The combination of CPF and PGNN achieves new state-of-the-art performance on MPII (PCKh@0.5) and competitive results on LSP with minimal parameter and FLOP overhead.
• Tree-like and loopy PGNN structures produce similar improvements, indicating robustness to graph topology.
• Propagation steps show diminishing returns after a small number (T≈3), with the first two steps contributing most to accuracy.
• Backbone experiments show ResNet-50 and Hourglass both benefit from CPF and PGNN, with Hourglass reaching 92.5% PCKh on MPII.