[Paper Review] You Only Look Twice: Rapid Multi-Scale Object Detection In Satellite Imagery

Adam Van Etten|arXiv (Cornell University)|May 24, 2018

Advanced Neural Network ApplicationsComputer Science4 references207 citations

TL;DR

YOLT adapts a fast, multi-scale, fully convolutional detector for very large overhead images, enabling near real-time localization of small objects like cars, airplanes, boats, buildings, and airports across massive satellite scenes.

ABSTRACT

Detection of small objects in large swaths of imagery is one of the primary problems in satellite imagery analytics. While object detection in ground-based imagery has benefited from research into new deep learning approaches, transitioning such technology to overhead imagery is nontrivial. Among the challenges is the sheer number of pixels and geographic extent per image: a single DigitalGlobe satellite image encompasses >64 km2 and over 250 million pixels. Another challenge is that objects of interest are minuscule (often only ~10 pixels in extent), which complicates traditional computer vision techniques. To address these issues, we propose a pipeline (You Only Look Twice, or YOLT) that evaluates satellite images of arbitrary size at a rate of >0.5 km2/s. The proposed approach can rapidly detect objects of vastly different scales with relatively little training data over multiple sensors. We evaluate large test images at native resolution, and yield scores of F1 > 0.8 for vehicle localization. We further explore resolution and object size requirements by systematically testing the pipeline at decreasing resolution, and conclude that objects only ~5 pixels in size can still be localized with high confidence. Code is available at https://github.com/CosmiQ/yolt.

Motivation & Objective

• Address the challenges of detecting very small objects in enormous satellite images.
• Develop a fast, dense-grid CNN architecture suitable for high-density, arbitrary-rotation overhead objects.
• Enable processing of images at native resolution by dividing large images into manageable chips and stitching results.
• Mitigate scale and rotation variance with data augmentation and multi-scale classifiers.
• Demonstrate transferability across sensors and analyze resolution dependence of detection performance.

Proposed method

• Extend the YOLO-inspired framework with a 22-layer dense network that downsamples by 16 and outputs a 26x26 prediction grid for 416x416 inputs.
• Introduce a passthrough layer that concatenates a high-resolution feature map to refine small object localization.
• Partition large images into overlapping cutouts, evaluate each with a detector, and stitch results into a global map.
• Apply non-maximum suppression on the global set of predictions to remove duplicate detections.
• Use dual classifiers at different scales to reduce confusion between small objects and large infrastructure (e.g., vehicles/buildings vs airports).
• Train with stochastic gradient descent using 5 boxes per grid, learning rate 1e-3, weight decay 0.0005, momentum 0.9.

Experimental results

Research questions

• RQ1 Can a YOLO-like detector be effectively adapted to overhead imagery with very small, densely packed objects?
• RQ2 Does a multi-scale (dual classifier) approach improve detection accuracy and reduce false positives for objects like airports vs vehicles/buildings?
• RQ3 How does object detection performance vary with ground sample distance (resolution) and object size in satellite imagery?
• RQ4 Is the approach transferable across sensors (e.g., DigitalGlobe to Planet) without extensive retraining?
• RQ5 What is the practical inference speed when processing arbitrarily large satellite images at native resolution?

Key findings

• YOLT achieves F1 scores in the range 0.61–0.91 across categories, with airports and vehicles performing strongest (e.g., airports F1 ≈ 0.91, cars ≈ 0.90).
• Inference speed is rapid on GPU, around 50 frames per second, with overall area-wide localization in minutes for city-scale regions.
• Dual-scale classifiers significantly improve results by avoiding confusion between small objects and large infrastructure, outperforming a single universal model.
• For cars, objects as small as ~5 pixels can be localized with high confidence, and performance degrades gradually with decreasing object size down to ~1 pixel.
• At 30 cm GSD, the method detects cars, airplanes, boats, buildings, and airports with varying F1 scores; airports show particularly high robustness across scales.
• The pipeline can localize vehicles and buildings at ~30 km^2 per minute and airports at ~6,000 km^2 per minute, suggesting feasibility for near-real-time satellite analytics.