[Paper Review] Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering

Pedro F. Proença|arXiv (Cornell University)|Jul 9, 2019

Space Satellite Systems and ControlEngineering34 references38 citations

TL;DR

The paper introduces URSO, a photorealistic Unreal Engine 4-based simulator, and a deep learning framework using probabilistic orientation soft classification to estimate 6D pose of known spacecraft from a single image, achieving high ESA challenge rankings (3rd synthetic, 2nd real).

ABSTRACT

On-orbit proximity operations in space rendezvous, docking and debris removal require precise and robust 6D pose estimation under a wide range of lighting conditions and against highly textured background, i.e., the Earth. This paper investigates leveraging deep learning and photorealistic rendering for monocular pose estimation of known uncooperative spacecrafts. We first present a simulator built on Unreal Engine 4, named URSO, to generate labeled images of spacecrafts orbiting the Earth, which can be used to train and evaluate neural networks. Secondly, we propose a deep learning framework for pose estimation based on orientation soft classification, which allows modelling orientation ambiguity as a mixture of Gaussians. This framework was evaluated both on URSO datasets and the ESA pose estimation challenge. In this competition, our best model achieved 3rd place on the synthetic test set and 2nd place on the real test set. Moreover, our results show the impact of several architectural and training aspects, and we demonstrate qualitatively how models learned on URSO datasets can perform on real images from space.

Motivation & Objective

• Motivate and enable monocular 6D pose estimation of known non-cooperative spacecraft under varied lighting and Earth backgrounds.
• Provide a photorealistic data generator (URSO) to train and evaluate DL models in space-like scenes.
• Propose a pose estimation framework using orientation soft classification to model angular ambiguity as a Gaussian mixture.
• Show sim-to-real transfer efficacy and analyze architectural/training factors that impact performance.

Proposed method

• Build URSO, a Unreal Engine 4-based simulator to render photorealistic spacecraft images with Earth textures and depth masks for labeled data.
• Adopt a ResNet-based backbone with a simplified end-to-end pose estimation head; use a relative translation loss plus an orientation term in the total loss.
• Propose orientation estimation via probabilistic soft classification over a discretized orientation space with Gaussian kernel encoding (K) and a weighted least-squares regression at test time.
• Introduce a multimodal orientation estimation via an EM framework fitting a Gaussian Mixture Model to recover multiple orientation hypotheses when views are ambiguous.
• Train with data augmentation including random camera perturbations and sim-to-real post-processing (grayscale conversion, exposure/contrast changes, AWG noise, blur, patch dropout).
• Evaluate on URSO datasets and the ESA SPEED benchmark, comparing orientation regression, soft classification, and multimodal approaches.

Experimental results

Research questions

• RQ1 Can a photorealistic Unity/UE4-based simulator provide labeled data that enables robust monocular 6D pose estimation of known spacecraft?
• RQ2 Is orientation soft classification (probabilistic, Gaussian-encoded) superior to direct quaternion regression for pose estimation under varying views and lighting?
• RQ3 How does data augmentation and sim-to-real transfer affect performance on real space imagery?
• RQ4 Can multimodal orientation estimation effectively resolve perceptual ambiguities in symmetric or ambiguous views?

Key findings

• URSO enables realistic synthetic data with labeled pose for training DL models for space pose estimation.
• Orientation soft classification outperforms direct orientation regression in angular accuracy.
• Modeling orientation with Gaussian mixtures (multimodal) improves handling of ambiguous views and yields better top-2 hypothesis detection.
• Data augmentation, especially camera perturbations and sim-to-real post-processing, significantly improves sim-to-real transfer and real-image accuracy.
• On the ESA SPEED benchmark, best single model achieved 0.1630 real error and 0.0604 synthetic error; a second model achieved 0.144 real and 0.067 synthetic, with a triple ensemble providing further gains (ranking details: 3rd on synthetic test set, 2nd on real test set).
• The approach shows sensitivity to image resolution and network depth, with 50-layer ResNet offering strong performance and higher resolutions aiding orientation estimation.