[Paper Review] WaveFill: A Wavelet-based Generation Network for Image Inpainting

Yingchen Yu, Fangneng Zhan+5 more|arXiv (Cornell University)|Jul 23, 2021

Advanced Image Processing Techniques46 references8 citations

TL;DR

WaveFill proposes a wavelet-based image inpainting network that decomposes images into frequency bands using discrete wavelet transform (DWT), enabling separate optimization of L1 loss for low-frequency structures and adversarial loss for high-frequency details. This frequency-disentangled approach mitigates inter-frequency conflicts, and a novel Frequency Region Attentive Normalization (FRAN) module ensures feature consistency across bands, achieving state-of-the-art performance in both qualitative and quantitative evaluations on multiple benchmarks.

ABSTRACT

Image inpainting aims to complete the missing or corrupted regions of images with realistic contents. The prevalent approaches adopt a hybrid objective of reconstruction and perceptual quality by using generative adversarial networks. However, the reconstruction loss and adversarial loss focus on synthesizing contents of different frequencies and simply applying them together often leads to inter-frequency conflicts and compromised inpainting. This paper presents WaveFill, a wavelet-based inpainting network that decomposes images into multiple frequency bands and fills the missing regions in each frequency band separately and explicitly. WaveFill decomposes images by using discrete wavelet transform (DWT) that preserves spatial information naturally. It applies L1 reconstruction loss to the decomposed low-frequency bands and adversarial loss to high-frequency bands, hence effectively mitigate inter-frequency conflicts while completing images in spatial domain. To address the inpainting inconsistency in different frequency bands and fuse features with distinct statistics, we design a novel normalization scheme that aligns and fuses the multi-frequency features effectively. Extensive experiments over multiple datasets show that WaveFill achieves superior image inpainting qualitatively and quantitatively.

Motivation & Objective

• To address the perception-distortion trade-off in image inpainting caused by concurrent optimization of reconstruction and adversarial losses in the spatial domain.
• To reduce inter-frequency conflicts arising when L1 and adversarial losses are applied jointly to entangled spatial features.
• To improve structural and texture consistency across frequency bands during image completion.
• To develop a normalization mechanism that aligns and fuses multi-frequency features effectively for artifact suppression and detail preservation.
• To achieve superior inpainting results on diverse datasets with both rectangular and free-form masks.

Proposed method

• The network uses 2D discrete wavelet transform (DWT) to decompose input images into low-frequency (LF), level-2 high-frequency (Lv2-HF), and level-1 high-frequency (Lv1-HF) bands, preserving spatial information.
• The low-frequency band is completed using L1 reconstruction loss via a gated residual block (GC ResBlock), while high-frequency bands are processed with adversarial loss to enhance texture realism.
• A novel Frequency Region Attentive Normalization (FRAN) module is introduced to attend from low to high frequencies, aligning and fusing multi-frequency features for consistent generation.
• Features from different frequency branches are propagated through FRAN, which computes attention maps based on frequency-specific statistics to guide feature fusion.
• The completed frequency bands are reconstructed into the spatial domain using inverse DWT (IDWT), producing the final inpainted image.
• The framework is trained end-to-end with a hybrid loss combining L1 loss on low-frequency bands and adversarial loss on high-frequency bands.

Experimental results

Research questions

• RQ1 Can separating the optimization of reconstruction and perceptual losses across frequency bands reduce inter-frequency conflicts in image inpainting?
• RQ2 How does wavelet-based frequency decomposition compare to spatial-domain or DCT-based methods in preserving structural and texture details?
• RQ3 Can a frequency-aware normalization scheme improve feature alignment and consistency across multi-frequency branches?
• RQ4 Does the proposed FRAN module effectively suppress artifacts and preserve fine-grained texture details during image completion?
• RQ5 To what extent does WaveFill outperform state-of-the-art methods on diverse inpainting benchmarks with varying mask types?

Key findings

• WaveFill achieves a Fréchet Inception Distance (FID) of 31.02 on the Paris StreetView validation set with irregular masks, outperforming the baseline (33.95) and DCT-based model (100.93).
• The model achieves a PSNR of 28.94 and SSIM of 0.904 on the same benchmark, demonstrating superior structural and perceptual quality.
• User studies show that WaveFill is judged as the most realistic inpainting in 68.0% of cases on Paris StreetView, 72.0% on Places2, and 70.0% on CelebA-HQ, significantly outperforming competitors.
• Abllation studies confirm that wavelet decomposition improves performance over DCT and spatial-domain baselines, with FID dropping from 100.93 (DCT) to 31.02 (WaveFill + FRAN).
• The FRAN module reduces artifacts and improves semantic consistency, as shown by visual comparisons where WaveFill produces fewer distortions and sharper textures than GC, GMCNN, and EC.
• The combination of wavelet decomposition and FRAN leads to a 2.5% improvement in FID and 0.5 dB gain in PSNR over the baseline model using feature concatenation.