[Paper Review] StructureFlow: Image Inpainting via Structure-aware Appearance Flow

Yurui Ren, Xiaoming Yu+4 more|arXiv (Cornell University)|Aug 11, 2019

Generative Adversarial Networks and Image SynthesisComputer Science35 references56 citations

TL;DR

StructureFlow splits image inpainting into structure reconstruction and texture generation, using edge-preserved smooth images for global structure and appearance flow with Gaussian sampling for texture synthesis guided by reconstructed structures.

ABSTRACT

Image inpainting techniques have shown significant improvements by using deep neural networks recently. However, most of them may either fail to reconstruct reasonable structures or restore fine-grained textures. In order to solve this problem, in this paper, we propose a two-stage model which splits the inpainting task into two parts: structure reconstruction and texture generation. In the first stage, edge-preserved smooth images are employed to train a structure reconstructor which completes the missing structures of the inputs. In the second stage, based on the reconstructed structures, a texture generator using appearance flow is designed to yield image details. Experiments on multiple publicly available datasets show the superior performance of the proposed network.

Motivation & Objective

• Improve global structure recovery in inpainting by using edge-preserved smooth images as structure guidance.
• Enable vivid texture synthesis by sampling features from regions with similar structures.
• Stabilize appearance flow training with Gaussian sampling and a sampling correctness loss.
• Demonstrate effectiveness on diverse datasets and analyze ablations to validate components.

Proposed method

• Two-stage architecture with a structure reconstructor Gs and a texture generator Gt.
• Structure reconstructor uses edge-preserved smooth images Se as guidance and minimizes L1 loss to Sgt with adversarial training.
• Texture generator uses appearance flow to warp and sample features from existing regions based on reconstructed structures, aided by a sampling correctness loss.
• Gaussian sampling replaces bilinear sampling in appearance flow to expand receptive field.
• Training includes L1, adversarial, and sampling-correctness loss terms with stage-wise pretraining and subsequent fine-tuning.

Experimental results

Research questions

• RQ1 Can edge-preserved smooth structures effectively guide global structure reconstruction for inpainting?
• RQ2 Does appearance flow guided texture generation with Gaussian sampling produce more realistic textures and coherent structures?
• RQ3 How do ablations of structure guidance, flow sampling, and loss terms affect inpainting quality on diverse datasets?

Key findings

• StructureFlow achieves competitive PSNR, SSIM, and FID on Places2 compared to CA, EdgeConnect, and PConv.
• Subjective user studies show StructureFlow outperforming competitors in highly structured scenes (Celeba and Paris) and remaining competitive on Places2.
• Ablation shows including the structure reconstructor improves performance; too little or too much smoothing (sigma) degrades results; appearance flow with Gaussian sampling and sampling-correctness loss yields better texture realism and flow stability.
• Gaussian sampling expands the receptive field of appearance flow, mitigating poor gradient propagation and enabling long-range feature matching.
• The sampling-correctness loss, based on VGG feature cosine similarities, helps constrain sampling to semantically relevant regions.