[Paper Review] Single-Image Lens Flare Removal.

Yicheng Wu, Qiurui He+5 more|arXiv (Cornell University)|Nov 25, 2020

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes a novel semi-synthetic pipeline that models lens flare using empirical and wave-optics simulations to generate realistic flare-corrupted images. Using this data, a neural network is trained to remove lens flare, achieving a 3dB PSNR improvement over state-of-the-art methods and generalizing well to real-world flares from diverse devices.

ABSTRACT

Lens flare is a common artifact in photographs occurring when the camera is pointed at a strong light source. It is caused by either multiple reflections within the lens or scattering due to scratches or dust on the lens, and may appear in a wide variety of patterns: halos, streaks, color bleeding, haze, etc. The diversity in its appearance makes flare removal extremely challenging. Existing software methods make strong assumptions about the artifacts' geometry or brightness, and thus only handle a small subset of flares. We take a principled approach to explicitly model the optical causes of flare, which leads to a novel semi-synthetic pipeline for generating flare-corrupted images from both empirical and wave-optics-simulated lens flares. Using the semi-synthetic data generated by this pipeline, we build a neural network to remove lens flare. Experiments show that our model generalizes well to real lens flares captured by different devices, and outperforms start-of-the-art methods by 3dB in PSNR.

Motivation & Objective

• Address the challenge of lens flare removal due to its highly variable appearance from geometric, brightness, and scattering effects.
• Overcome limitations of existing methods that rely on strong assumptions about flare geometry or intensity.
• Develop a physically grounded approach to simulate lens flares by modeling optical causes such as internal reflections and lens imperfections.
• Create a semi-synthetic data generation pipeline combining empirical measurements and wave-optics simulations for diverse flare patterns.
• Train a deep neural network on this synthetic data to achieve robust generalization to real-world lens flares across different devices.

Proposed method

• Model the optical origins of lens flare by simulating multiple reflections and scattering effects using wave-optics principles.
• Integrate empirical measurements from real lens systems to calibrate the simulated flare characteristics.
• Construct a semi-synthetic pipeline that combines real lens parameters with simulated flare patterns for data augmentation.
• Generate diverse, realistic flare-corrupted images by varying light source position, lens elements, and surface defects.
• Train a convolutional neural network end-to-end on the semi-synthetic dataset to predict flare-free images.
• Leverage the physical realism of the synthetic data to improve zero-shot generalization to real flares.

Experimental results

Research questions

• RQ1 Can a semi-synthetic data generation pipeline accurately simulate the diversity of real-world lens flare patterns?
• RQ2 Does modeling the physical causes of lens flare lead to better generalization in deep learning-based removal methods?
• RQ3 Can a model trained on semi-synthetic data effectively remove real lens flares from images captured by different devices?
• RQ4 How does the proposed method compare in performance to existing flare removal techniques under varying flare conditions?
• RQ5 To what extent does the use of wave-optics and empirical data improve the realism and effectiveness of synthetic training data?

Key findings

• The proposed semi-synthetic pipeline successfully generates realistic lens flare patterns that closely match real-world observations.
• The neural network trained on the semi-synthetic data achieves a 3dB PSNR gain over state-of-the-art methods on real flare removal benchmarks.
• The model generalizes effectively to real lens flares captured by different cameras and lenses, even without fine-tuning.
• The method outperforms existing approaches that rely on geometric or intensity-based assumptions, especially on complex flare patterns.
• The integration of wave-optics and empirical data significantly enhances the realism and diversity of synthetic training data.
• The approach demonstrates robustness to variations in flare type, including halos, streaks, and color bleeding.