[Paper Review] Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning

Supasorn Suwajanakorn, Noah Snavely+2 more|arXiv (Cornell University)|Jul 5, 2018

Human Pose and Action Recognition55 references132 citations

TL;DR

The paper presents KeypointNet, an end-to-end framework that learns a latent set of 3D keypoints and detectors optimized for downstream pose-estimation tasks, without ground-truth keypoint annotations, and demonstrates superior pose estimation performance over a supervised baseline on ShapeNet cars, chairs, and planes.

ABSTRACT

This paper presents KeypointNet, an end-to-end geometric reasoning framework to learn an optimal set of category-specific 3D keypoints, along with their detectors. Given a single image, KeypointNet extracts 3D keypoints that are optimized for a downstream task. We demonstrate this framework on 3D pose estimation by proposing a differentiable objective that seeks the optimal set of keypoints for recovering the relative pose between two views of an object. Our model discovers geometrically and semantically consistent keypoints across viewing angles and instances of an object category. Importantly, we find that our end-to-end framework using no ground-truth keypoint annotations outperforms a fully supervised baseline using the same neural network architecture on the task of pose estimation. The discovered 3D keypoints on the car, chair, and plane categories of ShapeNet are visualized at http://keypointnet.github.io/.

Motivation & Objective

• Motivate learning a minimal, category-specific set of 3D keypoints directly optimized for a downstream geometric task.
• Propose an end-to-end framework that jointly learns keypoint detectors and their 3D coordinates.
• Encourage keypoints that are geometrically consistent across views and object instances.
• Demonstrate that latent keypoints can outperform supervised keypoints in relative pose estimation.

Proposed method

• Define an end-to-end objective with a multi-view consistency loss and a relative pose estimation loss.
• Predict 3D keypoints (u,v,z) from single images using a translation-equivariant network with spatial softmax and depth prediction.
• Optimize keypoints via differentiable Procrustes (SVD) to align two views and compute pose error.
• Use a 3D separation loss and silhouette-consistency terms to ensure well-distributed, interior keypoints within silhouettes.
• Encode orientation information to break symmetry and stabilize keypoint assignment across symmetric objects.

Experimental results

Research questions

• RQ1 Can latent 3D keypoints learned end-to-end for a downstream task outperform manually annotated keypoints in pose estimation?
• RQ2 Do multi-view consistency and a differentiable pose objective yield geometrically meaningful and view-consistent keypoints across object instances?
• RQ3 How well do learned keypoints generalize across unseen shapes and viewpoints within ShapeNet categories?

Key findings

• KeypointNet learns geometrically and semantically meaningful 3D keypoints such as wheels, cockpit, wings, and chair legs.
• The unsupervised end-to-end method outperforms the supervised baseline on relative 3D pose estimation.
• Keypoints are consistent across views and unseen instances, and can infer depth without explicit depth supervision.
• Orientation prediction aids performance; even with occlusions, keypoints remain consistent across views.
• The approach generalizes to cars, chairs, and planes in ShapeNet and remains robust to large viewpoint changes.