[Paper Review] EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

Kamyar Nazeri, Eric Ng+3 more|arXiv (Cornell University)|Jan 1, 2019

Generative Adversarial Networks and Image Synthesis53 references583 citations

TL;DR

EdgeConnect uses a two-stage GAN architecture that first hallucinates edges in missing regions and then completes the image conditioned on those edges, achieving state-of-the-art inpainting on CelebA, Places2, and Paris StreetView.

ABSTRACT

Over the last few years, deep learning techniques have yielded significant improvements in image inpainting. However, many of these techniques fail to reconstruct reasonable structures as they are commonly over-smoothed and/or blurry. This paper develops a new approach for image inpainting that does a better job of reproducing filled regions exhibiting fine details. We propose a two-stage adversarial model EdgeConnect that comprises of an edge generator followed by an image completion network. The edge generator hallucinates edges of the missing region (both regular and irregular) of the image, and the image completion network fills in the missing regions using hallucinated edges as a priori. We evaluate our model end-to-end over the publicly available datasets CelebA, Places2, and Paris StreetView, and show that it outperforms current state-of-the-art techniques quantitatively and qualitatively. Code and models available at: https://github.com/knazeri/edge-connect

Motivation & Objective

• Motivate improved inpainting by preserving fine structural details through edge-guided reconstruction.
• Decouple high-frequency edge recovery from low-frequency color/textures via a two-stage network.
• Demonstrate end-to-end training and assess performance on standard benchmarks.
• Provide qualitative and quantitative evaluations showing superiority over prior methods.

Proposed method

• Two-stage adversarial model with an edge generator G1 and an image completion network G2.
• G1 hallucinates the missing region edges using masked grayscale input and an edge prior, trained with adversarial and feature-matching losses.
• Canny edge maps are used as training labels for edge generation, with spectral normalization applied to stabilize training.
• G2 completes the image by conditioning on a composite edge map combining ground-truth background edges with G1-generated edges, optimized with L1, adversarial, perceptual, and style losses.
• Discriminators use PatchGAN (70x70) with instance normalization; end-to-end fine-tuning without D1 for final refinement.
• Edge information is shown to improve inpainting quality; σ parameter of Canny detector controls edge amount and affects results.

Experimental results

Research questions

• RQ1 Does conditioning image inpainting on hallucinated edges improve the preservation of fine structures and reduce blur?
• RQ2 How does the amount and quality of edge information (via Canny σ) affect inpainting performance across datasets?
• RQ3 Can an end-to-end edge-guided inpainting model outperform existing state-of-the-art methods on standard benchmarks?
• RQ4 What is the impact of different edge representations (Canny vs. HED) on results?
• RQ5 How does the proposed EdgeConnect perform on irregular vs. regular masks and large missing regions?

Key findings

• EdgeConnect achieves superior quantitative results (lower L1, higher SSIM/PSNR, lower FID) on Places2 compared to CA, GLCIC, and PConv, and demonstrates qualitative improvements.
• Using edge information in the inpainting process leads to better structure preservation and reduced blur relative to edge-free baselines.
• A single σ ≈ 2 for Canny edges yields the best balance of edge detail and coverage for edge generation and subsequent inpainting.
• Ablation shows that including edge information (G2 with edges) consistently improves metrics across CelebA and Places2 datasets.
• The method handles irregular masks and multiple objects, and can be used for interactive editing (object removal) by manipulating edge maps.