[Paper Review] HighRes-net: Recursive Fusion for Multi-Frame Super-Resolution of Satellite Imagery

Michel Deudon, Freddie Kalaitzis+8 more|arXiv (Cornell University)|Feb 15, 2020

Advanced Image Processing Techniques42 references72 citations

TL;DR

HighRes-net introduces an end-to-end multi-frame super-resolution model that jointly learns co-registration, fusion, up-sampling, and a registered loss for satellite imagery, achieving top results on ESA’s Kelvin competition.

ABSTRACT

Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views. This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery. To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) registration-at-the-loss. Co-registration of low-resolution views is learned implicitly through a reference-frame channel, with no explicit registration mechanism. We learn a global fusion operator that is applied recursively on an arbitrary number of low-resolution pairs. We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet. We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth Observation data at scale. Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

Motivation & Objective

• Motivate grounded, reliable enhancement of satellite imagery through MFSR to assist environmental monitoring and policy actions.
• Develop an end-to-end architecture that handles co-registration and fusion of arbitrary numbers of LR frames.
• Introduce a registered loss via ShiftNet-Lanczos to align SR outputs with HR ground truth during training.
• Demonstrate state-of-the-art performance on real-world PROBA-V satellite data and competition benchmarks.

Proposed method

• Encode each LR view with a shared reference frame to enable implicit co-registration.
• Fuse encoded representations recursively to produce a single global latent, supporting variable numbers of views.
• Decode and upsample the fused latent to generate the super-resolved image.
• Use ShiftNet-Lanczos to learn sub-pixel shifts for a differentiable, registered loss during end-to-end training.
• Train end-to-end to minimize a joint loss combining a registered reconstruction loss and a regularized shift penalty.

Experimental results

Research questions

• RQ1 Can an end-to-end deep architecture learn co-registration and fusion for MFSR from an arbitrary set of LR views?
• RQ2 Does incorporating a registered loss via ShiftNet-Lanczos improve SR quality over unregistered losses?
• RQ3 How does HighRes-net perform on real satellite data compared to baselines in the ESA Kelvin competition?
• RQ4 What is the impact of the number of input views on reconstruction quality and generalization?

Key findings

• HighRes-net with ShiftNet-Lanczos achieved top scores on ESA Kelvin public and final leaderboards.
• The approach jointly learns fusion and registration, enabling end-to-end optimization for MFSR.
• An implicit co-registration mechanism via a reference frame improves fusion without explicit registration modules.
• End-to-end training with registered loss yields sharper, more accurate SR outputs than baselines.
• Training on PROBA-V real-world data avoids synthetic-downsampling biases and demonstrates scalability and efficiency.