[Paper Review] A Convolution Tree with Deconvolution Branches: Exploiting Geometric Relationships for Single Shot Keypoint Detection

Amit Kumar, Rama Chellappa|arXiv (Cornell University)|Apr 6, 2017

Face recognition and analysis26 references3 citations

TL;DR

This paper proposes a novel end-to-end, single-shot convolutional-deconvolution network with a pose-conditioned routing mechanism to improve facial keypoint detection by modeling geometric relationships between keypoints through learnable transform functions. The method uses a deconvolutional tree structure with pose-based routing to selectively propagate messages between visible and invisible keypoints, achieving state-of-the-art performance on AFLW and AFW datasets with 3.93 and 3.28 normalized mean error, respectively.

ABSTRACT

Recently, Deep Convolution Networks (DCNNs) have been applied to the task of face alignment and have shown potential for learning improved feature representations. Although deeper layers can capture abstract concepts like pose, it is difficult to capture the geometric relationships among the keypoints in DCNNs. In this paper, we propose a novel convolution-deconvolution network for facial keypoint detection. Our model predicts the 2D locations of the keypoints and their individual visibility along with 3D head pose, while exploiting the spatial relationships among different keypoints. Different from existing approaches of modeling these relationships, we propose learnable transform functions which captures the relationships between keypoints at feature level. However, due to extensive variations in pose, not all of these relationships act at once, and hence we propose, a pose-based routing function which implicitly models the active relationships. Both transform functions and the routing function are implemented through convolutions in a multi-task framework. Our approach presents a single-shot keypoint detection method, making it different from many existing cascade regression-based methods. We also show that learning these relationships significantly improve the accuracy of keypoint detections for in-the-wild face images from challenging datasets such as AFW and AFLW.

Motivation & Objective

• To address the challenge of accurate facial keypoint detection in unconstrained, in-the-wild images with extreme pose variations, occlusions, and missing annotations for invisible keypoints.
• To model geometric relationships between facial keypoints beyond score maps, by learning spatial dependencies at the feature level.
• To enable effective message passing between keypoints even when some are occluded or invisible, by conditioning the routing on estimated 3D head pose.
• To develop a fast, non-iterative, fully convolutional framework that avoids cascade regression and reduces model parameters via Residual SqueezeNet.
• To improve keypoint detection accuracy while simultaneously estimating 3D head pose in a unified multi-task learning framework.

Proposed method

• The method employs a deep convolutional network with deconvolutional branches forming a tree structure to propagate feature-level messages between facial keypoints.
• A pose-based routing function dynamically activates only relevant deconvolution branches based on the predicted 3D head pose, ensuring only meaningful messages are passed.
• Learnable transform functions, implemented as convolutions, model spatial relationships between keypoint pairs at the feature level, capturing geometric constraints.
• The network is trained end-to-end in a multi-task framework to predict 2D keypoint locations, visibility states, and 3D head pose simultaneously.
• Residual SqueezeNet is used to reduce model parameters while maintaining performance, enabling efficient inference.
• The framework implicitly handles occlusions by treating invisible keypoints as background during training, yet still predicts their locations accurately through message passing.

Experimental results

Research questions

• RQ1 Can geometric relationships between facial keypoints be effectively modeled at the feature level using learnable convolutional transforms in a deep network?
• RQ2 How can message passing between keypoints be made robust to occlusions and pose variations in unconstrained face images?
• RQ3 Can a single-shot, end-to-end deep learning framework outperform cascade regression methods in facial keypoint detection without 3D supervision?
• RQ4 To what extent does pose-conditioned routing improve keypoint detection accuracy compared to fixed or unconditioned message passing?
• RQ5 Can a unified network simultaneously predict 2D keypoint locations, visibility, and 3D head pose with high accuracy?

Key findings

• The proposed method achieves a normalized mean error (NME) of 3.93 on the AFLW dataset, outperforming all prior state-of-the-art methods except CCL, which uses reannotated data.
• On the AFW dataset, the method achieves an NME of 3.28, ranking among the best, with strong performance even on challenging partial and profile faces.
• The cumulative error distribution curves show that over 90% of faces on AFLW have an error below 5% of the face size, indicating robustness to variation.
• Qualitative results demonstrate accurate prediction of invisible keypoints, even in profile faces, due to effective message passing through the deconvolutional tree.
• The pose estimation performance on AFW shows that over 85% of faces are predicted within ±15° of the true head pose, confirming the effectiveness of the 3D pose estimation branch.
• Ablation studies confirm that the combination of pose-conditioned routing and feature-level message passing significantly improves detection accuracy over baselines without these components.