[Paper Review] Deep Image Homography Estimation

Daniel DeTone, Tomasz Malisiewicz+1 more|arXiv (Cornell University)|Jun 13, 2016

Advanced Vision and ImagingComputer Science13 references165 citations

TL;DR

The paper presents HomographyNet, a deep CNN that directly estimates image-to-image homographies from stacked grayscale patches, with both regression and classification variants, trained end-to-end using synthetically generated data from MS-COCO.

ABSTRACT

We present a deep convolutional neural network for estimating the relative homography between a pair of images. Our feed-forward network has 10 layers, takes two stacked grayscale images as input, and produces an 8 degree of freedom homography which can be used to map the pixels from the first image to the second. We present two convolutional neural network architectures for HomographyNet: a regression network which directly estimates the real-valued homography parameters, and a classification network which produces a distribution over quantized homographies. We use a 4-point homography parameterization which maps the four corners from one image into the second image. Our networks are trained in an end-to-end fashion using warped MS-COCO images. Our approach works without the need for separate local feature detection and transformation estimation stages. Our deep models are compared to a traditional homography estimator based on ORB features and we highlight the scenarios where HomographyNet outperforms the traditional technique. We also describe a variety of applications powered by deep homography estimation, thus showcasing the flexibility of a deep learning approach.

Motivation & Objective

• Demonstrate that a deep convnet can estimate homographies without separate feature detection and RANSAC-style robustification.
• Propose a 4-point corner-parameterization to stabilize learning and make the problem well-conditioned.
• Show two network variants (regression and classification) for predicting homography parameters.
• Provide an end-to-end data generation pipeline to synthesize large labeled datasets from real images.
• Evaluate against traditional ORB+RANSAC methods and discuss practical applications of deep homography estimation.

Proposed method

• Use a VGG-style 8-layer CNN that takes a 128x128x2 input (two stacked grayscale patches) and outputs 8 parameters.
• Adopt the 4-point parameterization where H4point encodes the four corner displacements (Delta u_i, Delta v_i) for i=1..4.
• Train two architectures: a regression head outputting 8 real-valued numbers with L2 loss; a classification head with 21 quantization bins per corner and a softmax/cross-entropy loss.
• Generate training data by randomly cropping patches from MS-COCO and applying random projective transforms to create ground-truth H^{AB} in a 4-point framework.
• Augment data generation with optional occlusions and motion blur to improve robustness.
• Train on 500,000 patch pairs and evaluate on a Warped MS-COCO test set; compare against ORB+RANSAC and an identity baseline.

Experimental results

Research questions

• RQ1 Can a deep CNN directly estimate image-to-image homographies without explicit local feature detection and RANSAC refinement?
• RQ2 Does the 4-point corner parameterization improve training stability and performance for deep homography estimation?
• RQ3 How do regression and classification formulations compare in accuracy and utility (e.g., confidence estimation) for homography estimation?
• RQ4 What is the achievable speed and practicality of deploying such models for real-time or embedded systems?

Key findings

• The regression HomographyNet achieves the best mean average corner error among evaluated methods.
• The classification HomographyNet provides corner-wise confidence distributions useful for debugging and uncertainty estimation.
• The system runs at over 300 fps on an NVIDIA Titan X, enabling real-time or embedded deployment.
• The 4-point parameterization yields a well-behaved learning problem by aligning corner displacements across dimensions.
• End-to-end learning eliminates the need for separate corner detection and robust homography estimation steps.
• Training with synthetically generated data from a large real-image dataset is effective for learning robust homography estimation.