[Paper Review] Imaging through the Atmosphere using Turbulence Mitigation Transformer

Xingguang Zhang, Zhiyuan Mao+2 more|arXiv (Cornell University)|Jul 13, 2022

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes the Turbulence Mitigation Transformer (TMT), a deep learning framework that improves multi-frame atmospheric turbulence restoration by integrating physics-based degradation modeling, a novel temporal-channel joint attention mechanism for efficient feature extraction, and a fast, accurate turbulence simulator. TMT achieves state-of-the-art performance on real-world turbulence data, outperforming existing methods in generalization, speed, and memory efficiency, especially when trained on synthetic data alone.

ABSTRACT

Restoring images distorted by atmospheric turbulence is a ubiquitous problem in long-range imaging applications. While existing deep-learning-based methods have demonstrated promising results in specific testing conditions, they suffer from three limitations: (1) lack of generalization capability from synthetic training data to real turbulence data; (2) failure to scale, hence causing memory and speed challenges when extending the idea to a large number of frames; (3) lack of a fast and accurate simulator to generate data for training neural networks. In this paper, we introduce the turbulence mitigation transformer (TMT) that explicitly addresses these issues. TMT brings three contributions: Firstly, TMT explicitly uses turbulence physics by decoupling the turbulence degradation and introducing a multi-scale loss for removing distortion, thus improving effectiveness. Secondly, TMT presents a new attention module along the temporal axis to extract extra features efficiently, thus improving memory and speed. Thirdly, TMT introduces a new simulator based on the Fourier sampler, temporal correlation, and flexible kernel size, thus improving our capability to synthesize better training data. TMT outperforms state-of-the-art video restoration models, especially in generalizing from synthetic to real turbulence data. Code, videos, and datasets are available at \href{https://xg416.github.io/TMT}{https://xg416.github.io/TMT}.

Motivation & Objective

• To address the lack of generalization from synthetic to real turbulence data in deep learning-based image restoration.
• To overcome memory and speed limitations in multi-frame video restoration transformers.
• To develop a fast, accurate, and physically grounded simulator for large-scale training data generation.
• To design a neural network architecture that explicitly models turbulence physics through de-tilting and de-blurring decomposition.
• To enable blind restoration of real-world turbulence-degraded sequences using purely synthetic training data.

Proposed method

• TMT decomposes the restoration task into two stages: de-tilting to correct wavefront tilt and de-blurring to remove spatially varying blur, improving physical plausibility and generalization.
• It introduces a temporal-channel joint attention (TCJA) module that applies self-attention along the temporal axis, reducing memory usage and enabling longer effective frame sequences.
• The method employs a multi-scale loss function that supervises training at multiple resolutions, enhancing feature learning and robustness.
• A new turbulence simulator is proposed using a Fourier-based sampler for dense-field simulation, incorporating temporal correlation and flexible kernel sizes to improve realism and training data diversity.
• The simulator enables fast, accurate, and scalable generation of synthetic turbulence data without interpolation or heuristic approximations.
• TMT is trained in a supervised manner on synthetic data and evaluated on real-world sequences without fine-tuning, demonstrating strong zero-shot generalization.

Experimental results

Research questions

• RQ1 Can a deep learning model trained solely on synthetic turbulence data generalize effectively to real-world turbulence degradation?
• RQ2 How can attention mechanisms be redesigned to reduce memory consumption while maintaining performance in long video sequences?
• RQ3 What role does physical modeling of turbulence—specifically de-tilting and de-blurring—play in improving restoration accuracy and generalization?
• RQ4 Can a fast, physically accurate simulator be designed to generate diverse, high-quality training data for turbulence restoration?
• RQ5 How does the proposed multi-scale supervision and TCJA attention mechanism compare to standard video restoration transformers in terms of speed, memory, and performance?

Key findings

• TMT achieves a PSNR of 28.4543, SSIM of 0.8539, and LPIPS of 0.1640 on real-world turbulence sequences, outperforming state-of-the-art methods like VRT and TSRWGAN.
• The model generalizes well to real-world data without fine-tuning, demonstrating strong zero-shot capability despite being trained exclusively on synthetic data.
• The temporal-channel joint attention (TCJA) mechanism reduces memory usage and enables processing of longer frame sequences compared to standard video transformers.
• The proposed simulator generates high-fidelity, dense-field turbulence data with improved temporal correlation and kernel flexibility, enabling better training data diversity.
• TMT surpasses conventional methods like CLEAR [4] and Mao et al. [5] in visual quality, producing sharper, more natural reconstructions with better detail preservation.
• When fine-tuned on synthetic data, TSRWGAN shows significant improvement on real-world datasets, confirming the value of the proposed synthetic data distribution.