[논문 리뷰] HighRes-net: Recursive Fusion for Multi-Frame Super-Resolution of Satellite Imagery

Michel Deudon, Freddie Kalaitzis외 8명|arXiv (Cornell University)|2020. 02. 15.

Advanced Image Processing Techniques참고 문헌 42인용 수 72

한 줄 요약

HighRes-net은 위성 영상에 대해 공동으로 co-registration, fusion, up-sampling, 및 registered loss를 학습하는 엔드 투 엔드 다중 프레임 초해상도 모델을 도입하여 ESA의 Kelvin 대회에서 최상위 성과를 달성한다.

ABSTRACT

Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views. This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery. To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) registration-at-the-loss. Co-registration of low-resolution views is learned implicitly through a reference-frame channel, with no explicit registration mechanism. We learn a global fusion operator that is applied recursively on an arbitrary number of low-resolution pairs. We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet. We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth Observation data at scale. Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

연구 동기 및 목표

• 환경 모니터링 및 정책 조치를 지원하기 위해 MFSR를 통한 위성 영상의 타당하고 신뢰할 수 있는 향상을 촉진한다.
• 임의 수의 LR 프레임의 co-registration과 fusion을 처리하는 엔드 투 엔드 아키텍처를 개발한다.
• 학습 중 SR 출력이 HR ground truth와 정렬되도록 ShiftNet-Lanczos를 통한 registered loss를 도입한다.
• 실세계 PROBA-V 위성 데이터와 대회 벤치마크에서 최첨단 성능을 입증한다.

제안 방법

• Encode each LR view with a shared reference frame to enable implicit co-registration.
• Fuse encoded representations recursively to produce a single global latent, supporting variable numbers of views.
• Decode and upsample the fused latent to generate the super-resolved image.
• Use ShiftNet-Lanczos to learn sub-pixel shifts for a differentiable, registered loss during end-to-end training.
• Train end-to-end to minimize a joint loss combining a registered reconstruction loss and a regularized shift penalty.

실험 결과

연구 질문

• RQ1 Can an end-to-end deep architecture learn co-registration and fusion for MFSR from an arbitrary set of LR views?
• RQ2 Does incorporating a registered loss via ShiftNet-Lanczos improve SR quality over unregistered losses?
• RQ3 How does HighRes-net perform on real satellite data compared to baselines in the ESA Kelvin competition?
• RQ4 What is the impact of the number of input views on reconstruction quality and generalization?

주요 결과

• HighRes-net with ShiftNet-Lanczos achieved top scores on ESA Kelvin public and final leaderboards.
• The approach jointly learns fusion and registration, enabling end-to-end optimization for MFSR.
• An implicit co-registration mechanism via a reference frame improves fusion without explicit registration modules.
• End-to-end training with registered loss yields sharper, more accurate SR outputs than baselines.
• Training on PROBA-V real-world data avoids synthetic-downsampling biases and demonstrates scalability and efficiency.