[Paper Review] Learning Delicate Local Representations for Multi-Person Pose Estimation

Yuanhao Cai, Zhicheng Wang+8 more|arXiv (Cornell University)|Mar 9, 2020

Human Pose and Action Recognition32 references37 citations

TL;DR

The paper introduces Residual Steps Network (RSN) for efficient intra-level feature fusion to learn delicate local representations, plus Pose Refine Machine (PRM) to balance local/global output features, achieving state-of-the-art results on COCO and MPII without extra data.

ABSTRACT

In this paper, we propose a novel method called Residual Steps Network (RSN). RSN aggregates features with the same spatial size (Intra-level features) efficiently to obtain delicate local representations, which retain rich low-level spatial information and result in precise keypoint localization. Additionally, we observe the output features contribute differently to final performance. To tackle this problem, we propose an efficient attention mechanism - Pose Refine Machine (PRM) to make a trade-off between local and global representations in output features and further refine the keypoint locations. Our approach won the 1st place of COCO Keypoint Challenge 2019 and achieves state-of-the-art results on both COCO and MPII benchmarks, without using extra training data and pretrained model. Our single model achieves 78.6 on COCO test-dev, 93.0 on MPII test dataset. Ensembled models achieve 79.2 on COCO test-dev, 77.1 on COCO test-challenge dataset. The source code is publicly available for further research at https://github.com/caiyuanhao1998/RSN/

Motivation & Objective

• Motivation: improve keypoint localization by preserving delicate local spatial information within the same feature level.
• Goal: propose intra-level feature fusion to learn richer local representations for multi-person pose estimation.
• Aim: design an output feature reweighting mechanism (PRM) to balance local and global cues for better keypoint accuracy.
• Demonstrate that RSN+PRM achieves state-of-the-art results on COCO and MPII without extra data or pretrained models.

Proposed method

• Propose Residual Steps Network (RSN) to fuse intra-level features with dense element-wise sums inside Residual Steps Blocks (RSB).
• RSB splits features into four branches, applies conv1x1 and incremental conv3x3s, and fuses with dense connections to widen receptive-field coverage (up to 15).
• Introduce Pose Refine Machine (PRM) as an attention module at the output to rebalance local/global representations using a multi-path design with channel and spatial attention pathways.
• PRM uses a global pooling path (channel-wise) and a depth-wise 9x9 path (spatial) plus an identity path to compute f_out = K(f_in) ⊗ (1 + β ⊗ α).
• Train and evaluate RSN+PRM on COCO and MPII, comparing against ResNet, Res2Net, DenseNet baselines and HRNet family.
• Show that RSN achieves better performance than baselines at similar GFLOPs and provides efficient, accurate keypoint localization.

Experimental results

Research questions

• RQ1 Can intra-level fusion within the same resolution improve delicate local representations for better keypoint localization?
• RQ2 Does an attention-based reweighting (PRM) properly trade off local vs. global features to boost pose estimation accuracy?
• RQ3 How does RSN compare to DenseNet, Res2Net, OSNet in terms of accuracy, efficiency, and localization quality on COCO and MPII?
• RQ4 Is PRM beneficial across single-stage and multi-stage architectures and when replacing standard attention modules like SE-CBAM?
• RQ5 What are the performance and speed trade-offs of RSN relative to HRNet across CPU/GPU in pose estimation tasks?

Key findings

• RSN consistently improves AP over ResNet and Res2Net at comparable GFLOPs (e.g., RSN-18 +2.9 AP vs ResNet-18, RSN-50 +2.5 AP vs ResNet-50).
• RSN maintains efficiency and outperforms DenseNet and Res2Net as model capacity grows, keeping higher AP at larger GFLOPs.
• PRM improves single-stage and multi-stage networks, delivering AP gains over non-attention baselines (e.g., ResNet-18 with PRM +1.5 AP).
• On COCO test-dev, RSN-50 with 4x RSN-50 achieves 78.0 AP (single model) and 79.2 AP with RSN-50 ensemble on 384x288 inputs, without pretrained backbones.
• On MPII, RSN yields state-of-the-art 93.0% PCKh@0.5 mean with 4x RSN-50.
• RSN offers faster inference than HRNet at similar accuracy, achieving higher pps on GPU and better CPU performance.