[Paper Review] Numerical Coordinate Regression with Convolutional Neural Networks

Aiden Nibali, Zhen He+2 more|arXiv (Cornell University)|Jan 23, 2018

Human Pose and Action RecognitionComputer Science17 references186 citations

TL;DR

We introduce a differentiable spatial to numerical transform (DSNT) that converts heatmaps into coordinates without extra parameters, improving end-to-end training and inference speed for coordinate regression like pose estimation, and show DSNT often outperforms heatmap matching and fully connected approaches.

ABSTRACT

We study deep learning approaches to inferring numerical coordinates for points of interest in an input image. Existing convolutional neural network-based solutions to this problem either take a heatmap matching approach or regress to coordinates with a fully connected output layer. Neither of these approaches is ideal, since the former is not entirely differentiable, and the latter lacks inherent spatial generalization. We propose our differentiable spatial to numerical transform (DSNT) to fill this gap. The DSNT layer adds no trainable parameters, is fully differentiable, and exhibits good spatial generalization. Unlike heatmap matching, DSNT works well with low heatmap resolutions, so it can be dropped in as an output layer for a wide range of existing fully convolutional architectures. Consequently, DSNT offers a better trade-off between inference speed and prediction accuracy compared to existing techniques. When used to replace the popular heatmap matching approach used in almost all state-of-the-art methods for pose estimation, DSNT gives better prediction accuracy for all model architectures tested.

Motivation & Objective

• Address limitations of heatmap matching and fully connected coordinate outputs in CNNs for coordinate regression.
• Propose a differentiable, parameter-free DSNT layer that preserves spatial generalization and end-to-end differentiability.
• Evaluate DSNT across CNN architectures on MPII human pose data to assess accuracy and inference efficiency.
• Explore regularization strategies to encourage meaningful heatmap shapes and improve coordinate prediction.

Proposed method

• Define DSNT as a differentiable layer that takes a single-channel normalized heatmap and computes coordinates as the mean of a discrete bivariate distribution.
• Represent coordinates as a 2D expectation with X and Y coordinate grids, enabling sub-pixel precision and backpropagation.
• Use heatmap activation functions (softmax, abs, ReLU, sigmoid) to create a normalized heatmap; softmax is found to perform best.
• Train end-to-end with Euclidean coordinate loss, not heatmap loss, ensuring the loss directly targets coordinate accuracy.
• Introduce regularization terms (variance and distribution divergences like KL/JS) to shape the heatmap and improve accuracy.
• Compare DSNT with heatmap matching and fully connected outputs, across ResNet and stacked hourglass architectures, at multiple heatmap resolutions.

Experimental results

Research questions

• RQ1 Can DSNT provide end-to-end trainable coordinate regression with preserved spatial generalization?
• RQ2 Does DSNT outperform traditional heatmap matching and fully connected approaches across architectures and heatmap resolutions?
• RQ3 What regularization strategies best improve DSNT performance and heatmap quality?
• RQ4 How do DSNT-based models compare in accuracy and inference speed to state-of-the-art pose estimation architectures?

Key findings

• DSNT consistently outperforms heatmap matching and fully connected outputs across tested architectures on MPII human pose data.
• DSNT yields better accuracy than heatmap matching even at low heatmap resolutions (e.g., 7x7), and remains robust as resolution increases.
• Regularization, especially Jensen-Shannon distribution regularization, improves DSNT accuracy over vanilla DSNT, with target Gaussian parameters showing robustness.
• A ResNet-50 backbone with DSNT (28px heatmap) achieves competitive accuracy with significantly faster inference and lower memory usage than larger hourglass models.
• DSNT enables sub-pixel coordinate predictions and full backpropagation through the coordinate output, unlike argmax-based approaches.
• When compared to stacked hourglass models, DSNT-based ResNet approaches offer favorable speed/memory trade-offs with only modest reductions in accuracy.