[Paper Review] Self-Adaptively Learning to Demoire from Focused and Defocused Image Pairs

Lin Liu, Shanxin Yuan+4 more|arXiv (Cornell University)|Nov 3, 2020

Advanced Image Processing Techniques48 references4 citations

TL;DR

This paper proposes a self-adaptive, test-time learning method for demoiréing images using a focused moiré-affected image and a corresponding defocused moiré-free blur image as input. The method employs an iterative, jointly optimized U-Net-like network and a 3-layer fully connected blur kernel predictor to restore clean images without explicit training, outperforming state-of-the-art methods on both synthetic and real-world datasets, including a new benchmark with texture moiré patterns.

ABSTRACT

Moire artifacts are common in digital photography, resulting from the interference between high-frequency scene content and the color filter array of the camera. Existing deep learning-based demoireing methods trained on large scale datasets are limited in handling various complex moire patterns, and mainly focus on demoireing of photos taken of digital displays. Moreover, obtaining moire-free ground-truth in natural scenes is difficult but needed for training. In this paper, we propose a self-adaptive learning method for demoireing a high-frequency image, with the help of an additional defocused moire-free blur image. Given an image degraded with moire artifacts and a moire-free blur image, our network predicts a moire-free clean image and a blur kernel with a self-adaptive strategy that does not require an explicit training stage, instead performing test-time adaptation. Our model has two sub-networks and works iteratively. During each iteration, one sub-network takes the moire image as input, removing moire patterns and restoring image details, and the other sub-network estimates the blur kernel from the blur image. The two sub-networks are jointly optimized. Extensive experiments demonstrate that our method outperforms state-of-the-art methods and can produce high-quality demoired results. It can generalize well to the task of removing moire artifacts caused by display screens. In addition, we build a new moire dataset, including images with screen and texture moire artifacts. As far as we know, this is the first dataset with real texture moire patterns.

Motivation & Objective

• To address the challenge of removing complex, variable moiré patterns from natural images and digital displays, where supervised methods struggle due to lack of diverse, high-quality training data.
• To enable effective demoiréing without requiring explicit training, by leveraging a defocused moiré-free image as a guide during test-time adaptation.
• To create a new benchmark dataset containing real-world moiré patterns from both digital screens and high-frequency textures, which is the first of its kind.
• To develop a method that is hardware-agnostic and applicable to standard digital cameras and smartphones without modifying existing capture pipelines.

Proposed method

• The method uses a dual sub-network architecture: one for image demoiréing (U-Net-like encoder-decoder) and one for blur kernel estimation (3-layer fully connected network), both optimized iteratively at test time.
• The demoiréing network takes the focused moiré image as input and predicts a clean, moiré-free image through iterative refinement.
• The blur kernel estimation network uses the defocused moiré-free image to predict a spatially varying blur kernel, modeling the low-pass filtering effect.
• The two networks are jointly optimized in an iterative, self-adaptive manner, with no pre-trained weights, enabling adaptation to unseen moiré patterns.
• The defocused image is generated via software or hardware defocusing, simulating a low-pass filter to remove moiré artifacts while preserving structural details.
• The method treats demoiréing as a joint filtering problem, where the blur kernel guides the restoration process, improving fidelity and detail recovery.

Experimental results

Research questions

• RQ1 Can a self-adaptive, test-time learning approach effectively remove complex moiré patterns without requiring explicit training on large-scale datasets?
• RQ2 Can a defocused moiré-free image serve as a reliable guide for restoring clean images from focused moiré-affected images?
• RQ3 How does the proposed method generalize to both screen-induced and texture-induced moiré patterns, especially in real-world scenarios?
• RQ4 Can a joint optimization of image restoration and blur kernel estimation outperform existing supervised and unsupervised demoiréing methods?

Key findings

• The proposed method, FDNet, achieves state-of-the-art performance on both public datasets (TIP2018, LCDMoire) and the newly introduced RealScreenMoire and RealTextureMoire datasets.
• On the RealTextureMoire dataset, which contains real high-frequency texture moiré patterns, FDNet significantly outperforms supervised methods like MopNet and SVLRM, which were trained on screen-specific data.
• Quantitative results show FDNet achieves higher PSNR and SSIM than competing methods on both synthetic and real-world test sets, indicating superior restoration quality and detail preservation.
• Visual comparisons demonstrate that FDNet effectively removes moiré patterns while maintaining sharp textures, unlike methods such as DIP and DoubleDIP, which leave residual artifacts or over-smooth details.
• The method generalizes well to screen-based moiré patterns, outperforming both unsupervised and supervised baselines on the SynScreenMoire and SynTextureMoire benchmarks.
• The ablation study confirms that the joint optimization of image restoration and blur kernel estimation is essential for performance, with ablation variants showing significant degradation when components are removed.