[Paper Review] Real-Time, Flight-Ready, Non-Cooperative Spacecraft Pose Estimation Using Monocular Imagery

Kevin P. Black, Shrivu Shankar+4 more|arXiv (Cornell University)|Jan 23, 2021

Space Satellite Systems and Control28 references22 citations

TL;DR

This paper presents a real-time, flight-ready monocular pose estimation system for non-cooperative spacecraft using a lightweight convolutional neural network (CNN) architecture trained exclusively on synthetic data. The system achieves state-of-the-art accuracy on real in-space imagery of the Northrop Grumman Enhanced Cygnus spacecraft and runs at 6.6 Hz on low-power flight-like hardware, demonstrating robust generalization and real-time performance without real training data.

ABSTRACT

A key requirement for autonomous on-orbit proximity operations is the estimation of a target spacecraft's relative pose (position and orientation). It is desirable to employ monocular cameras for this problem due to their low cost, weight, and power requirements. This work presents a novel convolutional neural network (CNN)-based monocular pose estimation system that achieves state-of-the-art accuracy with low computational demand. In combination with a Blender-based synthetic data generation scheme, the system demonstrates the ability to generalize from purely synthetic training data to real in-space imagery of the Northrop Grumman Enhanced Cygnus spacecraft. Additionally, the system achieves real-time performance on low-power flight-like hardware.

Motivation & Objective

• To develop a real-time, low-computational-cost pose estimation system suitable for small satellites with strict power and mass constraints.
• To enable accurate pose estimation of non-cooperative spacecraft using only synthetic training data, avoiding the scarcity of real in-space labeled images.
• To ensure the system runs in real-time on low-power, flight-like hardware such as the Intel Joule 570x single-board computer.
• To improve robustness to challenging space imaging conditions, including low signal-to-noise ratio, variable lighting, and complex backgrounds.
• To integrate an error prediction module that filters out unreliable pose estimates, enhancing system reliability.

Proposed method

• The system uses a three-stage CNN pipeline: object detection, keypoint regression, and perspective-n-point (PnP) pose solving.
• A novel error prediction network estimates uncertainty and flags poor-quality pose estimates for rejection.
• Synthetic images are generated using Blender’s Cycles engine with realistic degradations such as noise, blur, and lighting variations to improve generalization.
• The entire model is trained end-to-end on synthetic data of the Northrop Grumman Enhanced Cygnus spacecraft, with no real images used during training.
• Model inference is optimized using OpenVINO for x86 architecture, significantly improving speed on low-power processors.
• Hardware performance is benchmarked on the Intel Joule 570x, a flight-proven COTS single-board computer used in the NASA Seeker CubeSat mission.

Experimental results

Research questions

• RQ1 Can a monocular pose estimation system trained solely on synthetic data generalize effectively to real in-space imagery of a non-cooperative spacecraft?
• RQ2 Can a lightweight CNN-based architecture achieve real-time performance (≥1 Hz) on low-power, radiation-sensitive flight hardware?
• RQ3 How effective is the integration of an uncertainty-aware error prediction module in filtering out incorrect pose estimates?
• RQ4 To what extent does synthetic data augmentation with realistic image degradations improve robustness to real-world space imaging conditions?
• RQ5 What is the achievable inference speed and power consumption of a pose estimation system on flight-representative hardware like the Intel Joule 570x?

Key findings

• The system achieves state-of-the-art accuracy on the SPEED dataset, demonstrating strong performance on synthetic data of the ESA Tango spacecraft.
• Despite no real training data, the system generalizes effectively to real images of the Northrop Grumman Enhanced Cygnus spacecraft, with a median rotation error of 17.5° and translation error of 0.105 m in challenging viewing conditions.
• The system runs at 6.6 Hz on the Intel Joule 570x single-board computer after OpenVINO optimization, achieving real-time performance on flight-representative hardware.
• The error prediction network successfully identifies and rejects poor estimates, with one failure case showing a predicted error of 86, indicating high uncertainty.
• Inference time is reduced from 1.777 seconds (TensorFlow) to 0.152 seconds (OpenVINO), a 10x speedup, with consistent 3.7W power draw at thermal equilibrium.
• The system demonstrates robustness to rotational symmetry issues and complex backgrounds, with performance maintained even when key visual features like logos are obscured.